Summary of the Amazon S3 Service Disruption - <AUTHOR> https://aws.amazon.com/message/41926/ <comment><USER2> > _At 9:37AM PST, an authorized S3 team member using an established playbook > executed a command which was intended to remove a small number of servers > for one of the S3 subsystems that is used by the S3 billing process. > Unfortunately, one of the inputs to the command was entered incorrectly and > a larger set of servers was removed than intended._

It remains amazing to me that even with all the layers of automation, the root cause of most serious deployment problems remain some variant of a fat fingered user.

<comment><USER3> Look at the language used though. This is saying very loudly "Look, this isn't the engineer's fault here". It's one thing I miss about Amazon's culture- not blaming people when system's fail.

The follow-up doesn't bullshit with "extra training to make sure no one does this again", it says (effectively) "we're going to make this impossible to happen again, even if someone makes a mistake".

<comment><USER4> Any time I see "we're going to train everyone better" or "we're going to fire the guy who did it", all I can read is "this will happen again". You can't actually solve the problem of user error with training, and it's good to see Amazon not playing that game.

<comment><USER5> What bothered me about running TrueTech is that customers would sometimes demand repercussions against employees for making mistakes.

Enter Frans Plugge. Whenever a customer would get into that mode we'd fire Frans. This was easy, simply because he didn't exist in the first place (his name was pulled from a skit by two Dutch comedians, bonus points if you know who and which skit).

This usually then caused the customer to recant on how he/she never meant for anybody to get fired...

It was a funny solution and we got away with it for years, for one because it was pretty rare to get customers that mad to begin with and for another because Frans never wrote any blog posts about it ;)

But I was always waiting for that call from the labor board to ask why we fired someone for who there was no record of employment.

<comment><USER6> It is unreasonable for me to think that company owners should have the spine to say, "We take the decision to fire someone very seriously. We'll take your comments under consideration, but we retain sole discretion over such decisions."

It irks me that businesses fire people because of pressure from clients or social media. But having never been the boss, I may be missing something.

<comment><USER7> One reason to like a facet of Japanese management culture: if a customer wants someone to rake over the coals you offer management, not employees.

Internal repercussions notwithstanding, externally the company is a united front. It cannot cause mistakes by luck, accident, or happenstance, because the world includes luck, accidents, and happenstance, so any user-visible error is ipso facto a failure of management.

<comment><USER8> apparently this is (or was) a job in japan. companies would hire what amounts to an actor to get screamed at by the angry customer, and pretend to get fired on the spot. rinse, repeat whenever such appeasement is required.

<comment><USER9> I know one person who does this for real estate developers. He gets involved in contentious projects early on, goes to community meetings, offers testimony before the city council, etc. When construction gets going and people inevitably get pissed about some aspect of the project, he gets publicly fired to deflect the blame while the project moves on. Have seen it happen on three different projects in two cities now and, somehow, nobody catches on.

<comment><USER10> I don't know how to describe this in a single word or phrase appropriately, but I think it is a "genius problem" to exist. Not a genius solution. I feel that the problem itself is impressive and rich in layers of human nature, local culture etc - but once you have such a problem any average person could come up with a similar solution, because it is obvious.

It's still mind blowing and very amusing that this is a thing in our world!

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER11> This part is also interesting:

 _> While this is an operation that we have relied on to maintain our systems since the launch of S3, we have not completely restarted the index subsystem or the placement subsystem in our larger regions for many years._

These sorts of things make me understand why the Netflix "Chaos Gorilla" style of operating is so important. As they say in this post:

 _> We build our systems with the assumption that things will occasionally fail_

Failure at every level has to be simulated pretty often to understand how to handle it, and it is a really difficult problem to solve well.

<comment><USER12> I think you meant "Chaos Monkey" [1].

[1] [https://github.com/Netflix/chaosmonkey](https://github.com/Netflix/chaosmonkey)

<comment><USER13> No, _Chaos Gorilla is similar to Chaos Monkey, but simulates an outage of an entire Amazon availability zone_

[http://techblog.netflix.com/2011/07/netflix-simian- army.html](http://techblog.netflix.com/2011/07/netflix-simian-army.html)

</comment></comment></comment><comment><USER14> > _From the beginning of this event until 11:37AM PST, we were unable to > update the individual services' status on the AWS Service Health Dashboard > (SHD) because of a dependency the SHD administration console has on Amazon > S3._

Ensuring that your status dashboard doesn't depend on the thing it's monitoring is probably the first thing you should think about when designing your status system. This doesn't fill me with confidence about how the rest of the system is designed, frankly...

<comment><USER15> I'm curious as to why their fix was to host the Service Health Dashboard on more AWS regions. It seems like the responsible thing to do is to host it entirely on a competitor's service. That way, it's very simple to know that the status page will work no matter what happens to you.

<comment><USER16> If they did host their status page on a competitor's service, then they'd be reliant on that service, which might backfire if the competitor's service goes down while Amazon's own systems stay up.

What they really need is failover capability, which can fire up the status page on a competitor's service (or maybe on a completely separate disaster recovery site site owned by Amazon) in case Amazon's own services go down.

I'm sure Amazon's architects and engineers are more than capable of designing and implementing such a robust system and recognizing its importance. So it puzzles me as to why it wasn't done.

<comment> angry-hacker You can use several hosts, have two subdomains so if one is not red responding, engineers and managers know there are two status pages. Heck, have two different domains for them as well if there are dns issues. Amzstatus1.com and 2. Not dependent on Amazon domain anymore either.

</comment></comment></comment></comment><comment><USER17> "I did."

That was CEO Robert Allen's response when the AT&T network collapsed [1] on January 15, 1990

He was asked who made the mistake.

I can't imagine any CEO now a days making a similar statement.

[1] [http://users.csc.calpoly.edu/~jdalbey/SWE/Papers/att_collaps...](http://users.csc.calpoly.edu/~jdalbey/SWE/Papers/att_collapse.html)

<comment><USER18> Fascinating! Do you know of any source that documents the claim that Robert Allen made this statement?

<comment><USER17> I'll try. It was a dark day.

We all watched the news and I recall him saying that. The specific quote I don't remember but it was something like " you can consider that I did." I think he was asked what will happen to the person that caused it and who is that person.

Everyone knew right away this had to be human error. Right away. Switches simply had too much redundancy.

It was big then and not sure if I can locate a video.

<comment><USER19> Not sure if there's a video, but I found an article with the exact quote:

> As far as our customers are concerned, I did it.

[http://www.upi.com/Archives/1990/01/16/ATT-pinpoints- cause-o...](http://www.upi.com/Archives/1990/01/16/ATT-pinpoints-cause-of- long-distance-line-crash/2432632466000/)

<comment><USER17> Thanks. I got the gist at least.

I admired him and his answer at the time. The culture was quite professional and blame really never existed.

</comment></comment></comment></comment></comment><comment><USER20> Not as interesting an explanation as I was hoping for. Someone accidentally typed "delete 100 nodes" instead of "delete 10 nodes" or something.

It sounds like the weakness in the process is that the tool they were using permitted destructive operations like that. The passage that stuck out to me: "in this instance, the tool used allowed too much capacity to be removed too quickly. We have modified this tool to remove capacity more slowly and added safeguards to prevent capacity from being removed when it will take any subsystem below its minimum required capacity level."

At the organizational level, I guess it wasn't rated as all that likely that someone would try to remove capacity that would take a subsystem below its minimum. Building in a safeguard now makes sense as this new data point probably indicates that the likelihood of accidental deletion is higher than they had estimated.

<comment><USER21> I always wonder about unintended consequences of this sort of thing. Like someday there will be a worm about to rampage through their servers and someone says, "take them all offline now!" and the answer is, "we can't because of the throttle safeguard we put in place after incident XYZ, it will be about 17 hours..."

<comment><USER20> By safeguard I meant (and I think Amazon means too) an extra step that is required by the user before they can do the action so they don't do it by accident. Not something that prevents it entirely. Like how an MMO requires you before you delete a character to type the character's name in a box that pops up before you can delete it. That's far outside the realm of usual user interface, but that's so if you are just trying to edit a character it's impossible to accidentally hit that delete key. An analogous system for Amazon that would have prevented this outage: delete 10 nodes, ok. Delete 100 nodes, box pops up saying 'To delete this many nodes you must type the following in to a message box: "I want to take down a dangerously large amount of nodes."'

<comment><USER22> It occurs to me that having to type the English version of the numbers would probably work in this scenario.

       s3-shutdown -c "one hundred fifty"

But something simpler like a --emergency flag or the more whimsical --shutitdownshutitalldown

<comment><USER23> I think the biggest problem with flags like --emergency is if they end up in daily use, such as git --force. Then, they are both sudo-level AND used without a lot of though.

<comment><USER22> I was in fact thinking of the --force problem which is why I went another way.

</comment></comment></comment></comment></comment></comment><comment><USER24> Take a moment to look at the construction of this report.

There is no easily readable timeline. It is not discoverable from anywhere outside of social media or directly searching for it. As far as I know, customers were not emailed about this - I certainly wasn't.

You're an important business, AWS. Burying outage retrospectives and live service health data is what I expect from a much smaller shop, not the leader in cloud computing. We should all demand better.

<comment><USER25> Also notably missing is the "we will automatically refund all affected customers" line that we'd expect from somebody who wants to provide excellent service.

A graphical illustration of the service dependencies they were talking about would have been nice as well.

<comment><USER26> I mean, it's in the SLA that they have to refund 10% for the billing period IIRC.

<comment><USER27> If you request it and provide evidence that they find compelling.

 _To receive a Service Credit, you must submit a claim by opening a case in the AWS Support Center. To be eligible, the credit request must be received by us by the end of the second billing cycle after which the incident occurred and must include:

the words "SLA Credit Request" in the subject line; the dates and times of each incident of non-zero Error Rates that you are claiming; and your request logs that document the errors and corroborate your claimed outage (any confidential or sensitive information in these logs should be removed or replaced with asterisks). If the Monthly Uptime Percentage applicable to the month of such request is confirmed by us and is less than the applicable Service Commitment, then we will issue the Service Credit to you within one billing cycle following the month in which your request is confirmed by us. Your failure to provide the request and other information as required above will disqualify you from receiving a Service Credit."_

<comment><USER28> > provide evidence that they find compelling

A link to their tweet about the status page not working because the building was burning down around it seems compelling.

</comment></comment></comment></comment></comment><comment><USER29> > At 9:37AM PST, an authorized S3 team member using an established playbook > executed a command which was intended to remove a small number of servers > for one of the S3 subsystems that is used by the S3 billing process. > Unfortunately, one of the inputs to the command was entered incorrectly and > a larger set of servers was removed than intended.

I find making errors on production when you think you're on staging are a big one for similar errors. One of the best things I ever did on one job was to change the deployment script so that when you deployed you would get a prompt saying "Are you sure you want to deploy to production? Type 'production' to confirm". This helped stop several "oh my god, no!" situations when you repeated previous commands without thinking. For cases where you need to use SSH as well (best avoided but not always practical), it helps to use different colours, login banners and prompts for the terminals.

<comment><USER30> "All teams have a test server. Some teams are fortunate enough to also have a separate production server."

<comment><USER29> > "All teams have a test server. Some teams are fortunate enough to also have > a separate production server."

That's what I meant about the SSH comment. Not every team has the automation or infrastructure that allows you to avoid SSH.

</comment></comment></comment><comment><USER31> " we have not completely restarted the index subsystem or the placement subsystem in our larger regions for many years. S3 has experienced massive growth over the last several years and the process of restarting these services and running the necessary safety checks to validate the integrity of the metadata took longer than expected"

This is analogous to "we needed to fsck, and nobody realized how long that would take".

<comment><USER32> I feel they are lucky it was back up so fast.

</comment></comment><comment><USER33> Oh, that interview question. "Tell me about something you broke in your last job"

<comment><USER34> Paged for "$HOSTNAME is down!"

Usual checks to access $HOSTNAME failed

Rushed to office at 6am before important process was about to run that needed that host.

Plugged in keyboard+monitor, dead screen, nothing.

Physically power-cycled server.

Stood in front of monitor+keyboard. It occurred to me it was taking longer than expected to show POST screen. About that time, I got a page saying $ACTUALHOSTNAME is down.

Walk around to the back of the racks. The monitor cable had come detached from the cable extender that I plugged into the server. I had never plugged the monitor in at all, just the extension.

The server wasn't down in the first place, it just lost a virtual interface, which I was paged for, and stupidly tested that virtual interface instead of the REAL name/IP.

And then I raced to the office just so that _I_ could cause an outage.

</comment></comment><comment><USER35> TLDR; Someone on the team ran a command by mistake that took everything down. Good, detailed description. It happens. Out of all of Amazon's offerings, I still love S3 the most.

<comment><USER36> "It happens" is the only reasonable takeaway you can get from a postmortem like this. My worry is that people read it and go "I am aghast that such a command can be run!" without knowing that little commands like that are run numerous times a day without incident.

The only thing I read in there and go "hmmm" is that it took quite that long for the S3 service to recover, and that the status page wasn't hosted on someone that doesn't have an S3 dependency. That's just a plain "doh" moment :)

<comment><USER37> People need to realize when they go to the cloud it's not that 'it happens', it's that it will happen, and you have no ability to do anything about it. Fact of life and risk management.

<comment><USER30> ... and it's a different risk from self-hosting, but self-hosting provides all sorts of similar issues (such as when you do this to yourself, the cost is now coming out of your pocket, not Amazon's, to employ software engineers to harden your scripts against making the same mistake twice).

<comment><USER38> Not to mention, Amazon is catching the long tail of cloud failure like Google is catching the long tail of search keywords. They can now say with a somewhat straight face - "You know all those scripts you run to keep everything up? We have figured out many, many more possible ways for them to fail than you probably ever will, and we have added more layers of safeguards than you can even imagine."

</comment></comment></comment></comment></comment><comment><USER39> "we have changed the SHD administration console to run across multiple AWS regions."

Dear Amazon: please lease a $25/month dedicated server to host your status page on.

<comment><USER40> If big cloud companies would host their status pages on each other's servers, that would be... actually pretty cool.

</comment></comment><comment><USER41> AWS partitions its services into isolated regions. This is great for reducing blast radius. Unfortunately, us-east-1 has many times more load than any other region. This means that scaling problems hit us-east-1 before any other region, and affect the largest slice of customers.

The lesson is that partitioning your service into isolated regions is not enough. You need to partition your load evenly, too. I can think of several ways to accomplish this:

1\. Adjust pricing to incentivize customers to move load away from overloaded regions. Amazon has historically done the opposite of this by offering cheaper prices in us-east-1.

2\. Calculate a good default region for each customer and show that in all documentation, in the AWS console, and in code examples.

3\. Provide tools to help customers choose the right region for their service. Example: [http://www.cloudping.info/](http://www.cloudping.info/) (shameless plug).

4\. Split the large regions into isolated partitions and allocate customers evenly across them. For example, split us-east-1 into 10 different isolated partitions. Each customer is assigned to a particular partition when they create their account. When they use services, they will use the instances of the services from their assigned partition.

</comment><comment><USER42> So this is the second high profile outage in the last month caused by a simple command line mistake.

> Unfortunately, one of the inputs to the command was entered incorrectly and > a larger set of servers was removed than intended.

If I would have guessed anyone could prevent mistakes like this from propagating it would be AWS. It points to just how easy it is to make these errors. I am sure that the SRE who made this mistake is amazing and competent and just had one bad moment.

While I hope that AWS would be as understanding as Gitlab, I doubt the outcome is the same.

<comment><USER43> Amazon has the wherewithal to not freaking publicly name their actually human employee, so I'd imagine their culture around outages is probably a lot more healthy.

<comment><USER42> Well to be fair, he named himself within his notes and did not object to the public nature of the disclosure. I agree with your sentiment though that names should not be included within postmortems in the general case.

</comment></comment></comment><comment> rkuykendall-com tl;dr: Engineer fat-fingered a command and shut everything down. Booting it back up took a long time. Then the backlog was huge, so getting back to normal took even longer. We made the command safer, and are gonna make stuff boot faster. Finally, we couldn't report any of this on the service status dashboard, because we're idiots, and the dashboard runs on AWS.

<comment><USER30> Everything except the "We're idiots" part I'd agree with.

Self-hosting your diagnostic tools is an easy mistake to make, and I've seen both startups and large, multi-decade-experienced companies make it.

<comment> rkuykendall-com Of course. Meant it as more of an "I just spent 10 minutes searching for my car keys while holding them, because I'm an idiot." No disrespect to the engineers.

</comment></comment></comment><comment><USER44> Overall, it's pretty amazing that the recovery was as fast as it was. Given the throughput of S3 API calls you can imagine the kind of capacity that's needed to do a full stop followed by a full start. Cold-starting a service when it has heavy traffic immediately pouring into it can be a nightmare.

It'd be very interesting to know what kind of tech they use at AWS to throttle or do circuit breaking to allow back-end services like the indexer to come up in a manageable way.

</comment><comment><USER45> Something that wasn't addressed -- there seems to be an architectural issue with ELB where ELBs with S3 access logs enabled had instances fail ELB health checks, presumably while the S3 API was returning 5XX. My load balancers in us-east-1 without access logs enabled were fine throughout this event. Has there been any word on this?

<comment><USER46> I think it comes down to how important your ELB logs are -- if they are important enough that you don't want to allow traffic without logs (i.e. if you're using them for some sort of auditing/compliance), then failing when it can't write the logs seems like the right choice.

<comment><USER45> Thanks, that is a fair perspective. In our case we're using ELB logs as a redundant trace and it isn't critical that our traffic stops if the access logs fail. It would be nice if this behavior became a toggle in ELB settings, but think we can set something up to disable access logs programatically if we start seeing S3 issues.

<comment><USER47> Good luck with this. We tried to make changes yesterday to mitigate impact but AWS console was also affected. Was hesitant to make API calls for the changes since we werent sure they would complete successfully given all the services we found actually depended on S3 internally.

</comment></comment></comment></comment><comment><USER48> Really pleased to see this, it's good to see an organisation that's being transparent (and maybe given us a little peek under the hood of how S3 is architected) and most importantly they seem quite humbled.

It would be easy for an arrogant organisation to fire or negatively impact the person that made the mistake, I hope Amazon don't fall into that trap and focus instead on learning from what happened, closing the book and move on.

</comment><comment><USER49> There are quite a few comments here ignoring the clarity that hindsight is giving them. Apparently the devops engineers commenting here have never fucked up.

<comment><USER50> On the contrary: I feel like Amazon is taking some flak _because_ everyone here has messed up before, and are surprised that engineers (seemingly lacking failure experience) were able to do what they did.

I wouldn't task a junior sysadmin a server deletion, would you? Nor could I ever consider someone _without_ a fuckup a senior ;)

</comment></comment><comment><USER51> This is a bit off topic. The use of the word "playbook" suggests to me that they use Ansible to help manage S3. I wonder if that is the case, or if it's just internal lingo that means "a script". Unless there is some other configuration management system that uses the word playbook that I'm not aware of.

<comment><USER52> "playbook" is a relatively common term for "documented step-by-step procedure for specific tasks". Effectively, a script with #!/bin/human at the top.

<comment><USER53> Also known as a runbook

<comment><USER54> Or if you want to get real old school a checklist.

</comment></comment></comment></comment><comment><USER55> What does everyone use S3 for?

I'm genuinely curious. As my experiments with it have left me disappointed with its performance, I'm just not sure what I could use it for. Store massive amounts of data that is infrequently accessed? Well, unfortunately the upload speed I got to the standard rating one was so abysmal it would take too much time to move the data there; and then I suspect the inverse would be pretty bad as well.

<comment><USER56> 1 scenario: if you run a website that has a lot of static content (multiple GB of images, css, js, etc) and you dont want your http server to be responsible for serving that content then you give it all to s3 and let them serve it for you.

<comment><USER55> What about the performance of serving it? Sounds like I would need to cache it myself, anyway.

<comment><USER57> Performance out of S3 is generally really good. However, if you're looking to say, serve up a global website and your content is in a single S3 region, then you can leverage CloudFront CDN to serve up those objects. CloudFront integrates seamlessly with S3, and you don't pay transfer charges between CloudFront and S3.

</comment></comment></comment></comment><comment><USER58> > (...) [W]e have not completely restarted the index subsystem or the > placement subsystem in our larger regions for many years. S3 has experienced > massive growth over the last several years and the process of restarting > these services and running the necessary safety checks to validate the > integrity of the metadata took longer than expected.

All those tweets saying "turn it off and back on again"?

"We accidentally turned it off, but it hasn't been turned it off for so long it took us hours to figure out how to turn it back on."

Poorly-presented jokes aside, this is rather concerning. The indexer and placement systems are SPOFs!! I mean, I'd _presume_ these subsystems had ultra-low-latency hot failover, but this says _they never restarted_ , and I wonder if AWS didn't simply invest a ton of magic pixie dust in making Absolutely Totally Sure™ the subsystems physically, literally never crashed in years. Impressive engineering but also very scary.

At least they've restarted it now.

And I'm guessing the current hires now know a _lot_ about the indexer and placer, which won't do any harm to the sharding effort (I presume this'll be being sharded quicksmart).

I wonder if all the approval guys just photocopied their signatures onto a run of blank forms, heheh.

<comment><USER59> I don't think you don't understand the architecture of the system, if you are describing the indexer as a SPOF.

The system is a collection of shards. If you replicate it to create a second shard, then you'll just have a large a system, which is still a single point of failure.

The index, by necessity, has to be able to answer the question 'this object exists' or 'this object doesn't exit' \- so it needs to have consensus.

<comment><USER58> Hrm.

My speculative presumption was going off the sole datapoint of " _we have not completely restarted the index subsystem or the placement subsystem in our larger regions for many years_ ". I'm not quite sure how to interpret " _restart_ " in this context, mostly due to lack of exposure or experience.

The report also says " _Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. The servers that were inadvertently removed supported two other S3 subsystems._ " So you're right, it looks like multiple servers were supporting these systems, which does make sense (especially considering the load they would have seen). Okay.

I guess I didn't quite think through the load requirements and thought these were single machines - which is certainly ludicrous thinking :) - and that's where I got the SPOF reasoning from.

You're very right though, these consensus systems must be built as bottlenecks in order to see everything.

And there aren't really any alternatives: "build extra indexers and placement systems!" just gives you "but what if _all_ of them get taken offline?" and "it can't leave the datacenter, it sees 100GB/s of throughput" (number taken out of thin air).

Good points.

</comment></comment></comment><comment><USER60> I'm surprised how transparent this is, I can find Amazon often a bit opaque when dealing with issues.

<comment><USER61> I've found them to be very opaque in most contexts, but for major outages (which have been rare), they do have a history of solid postmortems.

<comment><USER62> The public postmortem from the big DynamoDB outage in 2015(?) is a great example I think: [https://aws.amazon.com/message/5467D2/](https://aws.amazon.com/message/5467D2/)

</comment></comment></comment><comment><USER63> I keep being reminded of something I read recently that made me feel uneasy about google's cloud spanner [1]:

 _the most important one is that Spanner runs on Google's private network. Unlike most wide-area networks, and especially the public internet, Google controls the entire network and thus can ensure redundancy of hardware and paths, and can also control upgrades and operations in general. Fibers will still be cut, and equipment will fail, but the overall system remains quite robust. It also took years of operational improvements to get to this point. For much of the last decade, Google has improved its redundancy, its fault containment and, above all, its processes for evolution. We found that the network contributed less than 10% of Spanner's already rare outages._

But when it fails it's going to be epic!

[1] [https://cloudplatform.googleblog.com/2017/02/inside-Cloud- Sp...](https://cloudplatform.googleblog.com/2017/02/inside-Cloud-Spanner-and- the-CAP-Theorem.html)

<comment><USER30> Google takes active steps to confirm that their fallback systems for mitigating failures work.

[http://queue.acm.org/detail.cfm?id=2371516](http://queue.acm.org/detail.cfm?id=2371516)

</comment></comment><comment> St-Clock I am unpleasantly surprised that they do not mention why services that should be unrelated to S3 such as SES were impacted as well and what they are doing to reduce such dependencies.

From a software development perspective, it makes sense to reuse S3 and rely on it internally if you need object storage, but from an ops perspective, it means that S3 is now a single point of failure and that SES's reliability will always be capped by S3's reliability. From a customer perspective, the hard dependency between SES and S3 is not obvious and is disappointing.

The whole internet was talking about S3 when the AWS status dashboard did not show any outage, but very few people mentioned other services such as SES. Next time we encounter errors with SES, should we check for hints of S3 outage before everything else? Should we also check for EC2 outage?

<comment><USER64> _services that should be unrelated to S3 such as SES were impacted_

I don't think this is particularly surprising. I'd already pretty much assumed that, e.g., a package of code for a Lambda function would be housed in an S3 bucket somewhere.

What's really surprising to me is how many of those buckets appear to live in US-EAST-1, and aren't able to keep functioning in a catastrophe by failing over to a different region.

<comment><USER65> You specify the run region for each of those services and all the other components get restricted to those regions too.

We're in ap-southeast-2 (Sydney) and none of our services were impacted yesterday.

</comment></comment></comment><comment><USER66> Do you know if Amazon is giving any refunds/credits for the service outbreak?

<comment><USER67> I don't think it's automatic. I just helped my former boss with his decision to go for a refund (he asked me for help drafting a request, but I reminded him that like 99.99% of their S3 storage is backups that are IA-Standard, so it may not be worth it).

</comment></comment><comment><USER68> I wouldn't want to be the person who wrote the wrong command! Sheesh.

<comment><USER46> I brought down our production system after a typo in a command once... the dev team took the blame for allowing an illegal parameter to bring down the system.

<comment><USER69> Kudos to the dev team for that, I think most people wouldn't own that kind of issue

<comment><USER46> It was a very well-run engineering department where taking blame was not a career ending decision. I took full blame for the typo (at 3am trying to resolve a customer issue), but the dev team accepted full responsibility for letting it take down the system.

Every mistake was used as a learning opportunity to ensure that the same and similar mistakes can't be repeated.

</comment></comment></comment></comment><comment><USER70> It's curious they needed to "remove capacity" to cure a slow billing problem.

Is that code for a "did you try to reboot the system?" kind of troubleshooting?

It sounds to me like the authorized engineer sent a command to reboot/reimage a large swath of the S3 infrastructure.

</comment><comment><USER71> If Amazon were a guy, he'd be a standup guy. This is a very detailed and responsible explanation. S3 has revolutionized my businesses and I love that service to no end. These problems happen very rarely but I may have backups just in case using nginx proxy approach at some point and because S3 is so good, everyone seems to adopt their API so its just a matter of a switch statement. Werner can sweat less. Props.

I would add, it would be awesome if there was a simulation environment, beyond just a test environment that simulated servers outside requesting in, before a command was allowed to run onto production, like a robot deciding this, then could mitigate this, kind of like TDD on steriods if they don't have that already.

</comment><comment><USER72> Imagine being THAT guy.......... in that exact moment...... after hitting enter and realizing what he did. RIP

<comment><USER73> I can imagine being that guy in that exact moment. But I can't imagine being that guy after the event. There will be a constant fear and doubt in my mind. And a constant fear whether others trust me anymore. I couldn't quit because that might make me look bad and I couldn't continue because that might make me look bad.

<comment><USER74> This guy may actually need therapy to not suffer from some light degree of trauma.

</comment></comment></comment><comment><USER75> Ops and Engineering here.

My guts hurt just reading this.

With big failures is never just one thing. There are a series of mistakes, bad choices, and ignorance that lead to a big system wide failures.

</comment><comment><USER76> Twitter once had 2 hours of downtime because an operations engineer accidentally asked a tool to restart all memcached servers instead of a certain server. The tool was then changed to make sure that you couldn't restart more than a few servers without additional confirmation. Sounds very similar to this situation. Something to think about when you are building your tools to be more error proof.

<comment><USER39> There are very few things that Twitter hasn't had two hours of downtime because of.

</comment></comment><comment><USER77> > _Unfortunately, one of the inputs to the command was entered incorrectly and > a larger set of servers was removed than intended._

We have geo-distributed systems. Load balancing and automatic failover. We agonize over edge cases that might cause issues. We build robust systems.

At the end of he day reliability -- a lot like security -- is most affected by the human factor.

</comment><comment><USER78> > Removing a significant portion of the capacity caused each of these systems > to require a full restart.

I'd be interested to understand why a cold restart was needed in the first place. That seems like kind of a big deal. I can understand many reasons why it might be necessary, but that seems like one of the issues that's important to address.

<comment><USER25> Possibly a consensus algorithm that refuses writes when it detects itself in a minority, because it think it's in the smaller part of a split-brain scenario.

In this case, throwing away and then re-provisioning the split-off nodes is a viable approach.

</comment></comment><comment><USER79> I find this refreshingly candid; human, even, for AWS.

</comment><comment><USER80> I hope I never have to write a post-mortem that includes the phrase "blast radius"

</comment><comment><USER81> "we have not completely restarted the index subsystem or the placement subsystem in our larger regions for many years."

Yeah... nothing says "resilience" quite like that...

</comment><comment><USER82> It sounds like this can be mitigated by making sure everything is run in dry run mode first, and for something mission critical, getting it double-checked by someone before removing the dry run constraint.

It's good practice in general, and I'm kind of astonished it's not part of the operational procedures in AWS, as this would have quickly been caught and fixed before ever going out to production.

<comment><USER83> I'm not sure you've understood what the problem is. They were removing some servers from a group. This isn't something that get's a dry run, not without spinning up the _entire_ AWS infrastructure. It also wouldn't have helped a jot, since the issue came about after an employee executing a playbook made a typo.

There's no way this could have been mitigated with a dry run. They're mitigating it in future by putting more aggressive safeguards in their tooling, which is the correct way to mitigate this sort of issue.

</comment></comment><comment><USER84> "As a result, (personal experience and anecdotal evidence suggest that) for complex continuously available systems, Operations Error tends to be the weakest link in the uptime chain."

[https://zvzzt.wordpress.com/2012/08/16/a-note-on- uptime/](https://zvzzt.wordpress.com/2012/08/16/a-note-on-uptime/)

</comment><comment><USER85> Same as that db1 / db2 for GitLab, naming things is pretty important (E.g production / staging, production-us-east-1-db-560 etc.)

</comment><comment><USER86> I guess it is time to define commands whose inputs have great distance in, say the Damerau-Levenshtein metric.

For numerical inputs, one might use both the digits and the textual expression. This would make them quite cumbersone but much less prone to errors. Or devise some shorthand for them...

156 (on fi six). 35. (zer th fi). 170 (on se zer). 28 (two eig) evens have three letters odds have two.

This is just my 2 cents.

</comment><comment><USER87> Are customers going to receive any kind of future credit because of this? Would be a nice band-aid after such a hard smack on the head.

<comment><USER88> You can make a claim to receive credit based on their SLA: [https://aws.amazon.com/s3/sla/](https://aws.amazon.com/s3/sla/)

<comment><USER89> You shouldn't have to make a claim - this should be handled by them automatically.

<comment><USER87> Agreed

</comment></comment></comment></comment><comment><USER90> If this boils down to an engineer incorrectly entering a command can we please refer to this outage as "Fat Finger Tuesday"?

</comment><comment><USER91> "People make mistakes all the time...the problem was that our systems that were designed to recognize and correct human error failed us." [1]

[1] [http://articles.latimes.com/1999/oct/01/news/mn-17288](http://articles.latimes.com/1999/oct/01/news/mn-17288)

<comment><USER92> This reminds me of Asimov's characteristically tiny story "Fault-Intolerant" [https://unotices.com/book.php?id=38686&page=15](https://unotices.com/book.php?id=38686&page=15) (You can ignore the story at the top about Feghoot, the real story is below.)

</comment></comment><comment><USER93> Amazon's AWS Outage Was More of a Design(UX) Failure and Less of Human Error. [https://www.linkedin.com/pulse/how-small-typo-caused- massive...](https://www.linkedin.com/pulse/how-small-typo-caused-massive- downtime-s3aws-hemant-kumar-singh)

</comment><comment><USER94> Wonder if every numbers for critical command lines shouldn't be spelled out as well. If you think about how checks works, you're supposed to write the number as well as the words for the number. -nbs two_hundreds instead of twenty is much less likely to happen..

just like rm -rf / should really be rm -rf `root`

</comment><comment><USER95> So this week the poor soul at Amazon, along with the Price-Waterhouse guy, are the poster children of Human Error.

</comment><comment><USER96> > While this is an operation that we have relied on to maintain our systems > since the launch of S3, we have not completely restarted the index subsystem > or the placement subsystem in our larger regions for many years.

This is the bit that'd worry me most; you'd think they'd be testing this.

<comment><USER97> A complete restart of the index subsystem would require downtime. Note: they are not saying those servers have never been restarted - it's highly likely they get restarted regularly. But, a complete restart of the index subsystem implies that you shut everything down first and restart it all at once, which is what was forced to happen two days ago.

<comment><USER98> Why can't the index subsystem itself have a backup then? When the primary subsystem is being restarted/rebuilt, the secondary takes over.

<comment><USER99> thats kind of like asking why git doesn't have a single backup... it's a distributed system, there's not just one backup, there are lots of little partial backups.

</comment></comment></comment></comment><comment><USER100> This caused panic and chaos for a bit among my team, which I imagine was replicated across the web.

Moments like these always remind me that a particularly clever or nefarious set of individuals could shut down essential parts of the Internet with a few surgical incisions.

</comment><comment><USER101> Seems like something like Chaos monkey should have been able to predict and mitigate a issue like this. Im actually curious if anyone uses it at all- Curious if anyone in here at a large company (over 500 employees) has it deployed or not.

</comment><comment><USER77> Remember folks, automate your systems but never forget to add sanity checks.

</comment><comment><USER102> I think they should have led with insensitivity about it and maybe a white lie. Such as... We took our main region us-east-1 down for X hours because we wanted to remind people they need to design for failure of a region :-)

Shameless plugs (authored months ago): [http://tuxlabs.com/?p=380](http://tuxlabs.com/?p=380) \- How To: Maximize Availability Effeciently Using AWS Availability Zones ( note read it, its not just about AZ's it is very clear to state multi-regions and better yet multi- cloud segway...second article) [http://tuxlabs.com/?p=430](http://tuxlabs.com/?p=430) \- AWS, Google Cloud, Azure and the singularity of the future Internet

</comment><comment><USER103> This makes me want to write a program that would ask users to confirm commands if it thinks they are running a known playbook and deviating from it. Does anyone know if a tool like that exists?

<comment><USER104> Not sure, but my company's fleet wide root scripts confirm first the exact command you want to run, then run on 1 host first and output the the full logs for you to inspect/confirm, and then finally start the full fleet wide run after you have confirmed the expected result of your output. They also output the full logs of across the entire fleet once your fleet wide script is run.

</comment></comment><comment><USER105> For as much as people jumped all over Gitlab last month, this seems remarkably similar in terms of preparedness for accidental and unanticipated failure.

<comment><USER83> Beyond a typing mistake, it's not really very similar. The Gitlab incident was one avoidable problem after another, ending with a giant WTF when they found out that no-one had even tested the backups were working.

This is a case of someone slipping on the keyboard, removing more capacity than intended and the recovery process taking longer than expected. The process actually seems to be working (to a given value of working), but the amount of downtime was way above acceptable. They've already put more safeguards into the tooling to prevent the situation from happening again.

S3 is also orders of magnitude more complex than Gitlabs infrastructure, so while the amount of time the outage lasted for is not acceptable, it does show that they at least have working processes for critical situations that allow them to get back in service within a day, which is pretty impressive.

</comment></comment><comment><USER106> Deletions, shutdowns and replications should always either contain a SELECT to show affected entities or a confirmation (Y/n) of the step.

</comment><comment><USER107> This is the risk you run into by doing everything through the command line. This would be really hard to do through a good GUI.

<comment><USER48> I highly doubt this claim, humans make mistakes regardless of the control method.

In this particular case the scripts didn't have adequate protections in place, but that's the benefit of hindsight

<comment><USER107> Or the benefit of testing.

</comment></comment></comment><comment><USER108> I guess this means it's much better for your app to fail over between regions than between availability zones.

</comment><comment><USER109> Man, I'd really hate to be that guy.

</comment><comment><USER110> Can Amazon take responsibility and offer say 10% discount to all the customers who are spending >$X ?

<comment><USER111> It's already part of their SLA.

[https://aws.amazon.com/s3/sla/](https://aws.amazon.com/s3/sla/)

</comment></comment><comment><USER112> Wow, I'm sure that person who mis entered the command will never, ever, ever, do it again.

</comment><comment><USER113> I don't know if it's a C thing, but those code comments are art!

<comment><USER114> posting in wrong thread?

<comment><USER113> oops yeah, I had so many HN tabs.

</comment></comment></comment><comment><USER115> I'm just going to call this "PEBKAC at scale"

</comment><comment><USER116> Great shock to the quick recovery of such a big system.

</comment><comment><USER117> Sounds so chernobyl.

<comment><USER118> Because it was resolved in a few hours without any further fallout? (Pun definitely intended)

</comment></comment><comment><USER119> There should be some sort of GUI interface that does appropriate checks instead of allowing someone to mistakenly type the correct information.

</comment><comment><USER120> Did you read the fucking article?

That is EXACTLY what they are doing (among other things).

<comment><USER121> Please stop posting like this so we don't have to ban you.

[https://news.ycombinator.com/newsguidelines.html](https://news.ycombinator.com/newsguidelines.html)

We detached this comment from [https://news.ycombinator.com/item?id=13776335](https://news.ycombinator.com/item?id=13776335) and marked it off-topic.

</comment></comment><comment><USER122> TLDR:

Never type 'EXEC DeleteStuff ALL'

When you actually mean 'EXEC DeleteStuff SOME'

</comment><comment><USER123> Something doesn't pass the smell test. Over two hours to reboot the index hosts?

<comment><USER124> I assume "reboot" in this instance means more than turning it off and on again --it must return to a working state, with many volumes of data requiring log processing to find the last (and best) "good state".

</comment></comment><comment><USER125> They could've just not post anything. People already forgot about this disruption.

</comment><comment><USER126> No on in HN is questioning this - "The Amazon Simple Storage Service (S3) team was debugging an issue causing the S3 billing system to progress more slowly than expected." \- they are debugging on Production System..

</comment><comment><USER127> What most AWS customers don't realize is that AWS is poorly automated. Their reliability relies on exploiting the employees to manually operate the systems. The technical bar at Amazon is incredibly low and they can't retain any good engineers.

</comment><comment><USER128> What's missing is addressing the problems with their status page system, and how we all had to use Hacker News and other sources to confirm that US East was borked.

<comment><USER129> No, this is addressed:

>We understand that the SHD provides important visibility to our customers during operational events and we have changed the SHD administration console to run across multiple AWS regions.

<comment><USER130> Which is fine until those regions go down. A status page, in my mind, should have a fallback on a completely different service provider.

<comment><USER128> Exactly. What they should do is have the status system be independent of AWS so it can report issues regardless of AWS service status.

<comment><USER118> Awsnap! [http://dilbert.com/strip/2008-02-15](http://dilbert.com/strip/2008-02-15)

</comment></comment></comment></comment></comment><comment><USER102> I think they should have led with insensitivity about it and maybe a white lie. Such as... We took our main region us-east-1 down for X hours because we wanted to remind people they need to design for failure of a region :-)

Shameless plugs (authored months ago): [http://tuxlabs.com/?p=380](http://tuxlabs.com/?p=380) \- How To: Maximize Availability Effeciently Using AWS Availability Zones ( note read it, its not just about AZ's it is very clear to state multi-regions and better yet multi- cloud segway...second article)

[http://tuxlabs.com/?p=430](http://tuxlabs.com/?p=430) \- AWS, Google Cloud, Azure and the singularity of the future Internet

</comment><comment><USER131> For the many of us who have built businesses dependent on S3, is anyone else surprised at a few assumptions embedded here?

* "authorized S3 team member" \-- how did this team member acquire these elevated privs?

* Running playbooks is done by one member without a second set of eyes or approval?

* "we have not completely restarted the index subsystem or the placement subsystem in our larger regions for many years"

The good news:

* "The S3 team had planned further partitioning of the index subsystem later this year. We are reprioritizing that work to begin immediately."

The truly embarrassing that everyone has known about for years is the status page:

* "we were unable to update the individual services' status on the AWS Service Health Dashboard "

When there is a wildly-popular Chrome plugin to _fix_ your page ("Real AWS Status") you would think a company as responsive as AWS would have fixed this years ago.

<comment><USER132> >Unauthorized S3 team member

This is not in the post.</comment></comment>


GitUp - <AUTHOR> http://gitup.co/ <comment><USER2> I was going to ask why this is better than TortoiseGIT, but the animated gif showed that here you can edit the message of a commit easily. I hate to invoke a complete rebase only to fix a typo in the message.

Question/Feature request: Can I reorder commits?

<comment><USER3> You can re-order commits on the spot and a lot more. See overview of app at [http://gitup.co](http://gitup.co) and docs: "Editing the Repository" in [http://forums.gitup.co/t/using-gitup-map- view/34](http://forums.gitup.co/t/using-gitup-map-view/34).</comment></comment>


They Write the Right Stuff - <AUTHOR> http://www.fastcompany.com/node/28121/print <comment><USER2> It's been posted around a thousand times, and the standard criticism still holds: The space program represents a unique, unreproducible environment in which quality is a benefit that outweighs every possible cost.

<comment><USER3> It's crazy the amount of effort they put into the process, and yet the slightest computer bug blows up a whole shuttle. I forget which shuttle it was, but an arithmetic bug was replicated in both failsafe systems, and it was exactly this bug that caused the shuttle to crash. Nowhere is Murphy's law more true.

<comment><USER4> There have been two losses of shuttles: 1986 with the failure of the O-ring in the solid-rocket booster and 1996 with the foam that fell off during lift-off that damaged a tile on the leading-edge of a wing which caused the shuttle to burn up on re-entry. Which loss are you talking about when you say, <i>it was exactly this bug that caused the shuttle to crash</i>?

<comment><USER5> Not shuttle. Mars probe. And it was a bad unit conversion.

<comment><USER3> Yeah, that's the one.

</comment></comment></comment></comment></comment><comment><USER6> If normal software buyers were as averse to defects as NASA, all software development would probably be like this. However, normal customers tend to prefer more features and overlook defects (relatively speaking).

</comment><comment><USER7> "The answer is, yes, the process does _stifle creativity_. [...] It's the process that offers a template for any _creative_ enterprise that's looking for a method to produce consistent - and consistently improving -- quality."

Come again?...

"In the modern software environment, 80% of the cost of the software is spent after the software is written the first time -- _they don't get it right the first time_ , so they spend time flogging it."

I'd say most sofware _cannot_ be gotten right the first time...

</comment><comment><USER8> Cool story, but don't run your startup like that.

If you do, you will deliver perfect software, years too late, that nobody wants.

<comment><USER9> Not to say that every project needs to be planned like NASA, but I would like to know how many start-ups have calculated their acceptable rate of failure (in their software) prior to launch.

</comment></comment><comment><USER9> This is one of the coolest things I have ever read. Thanks to OP.</comment>


Our Startup Handbook - <AUTHOR> https://versionone.vc/startup-handbook/ <comment><USER3> > It's better to use LinkedIn as a source for candidates, but when possible, > send a short and personal email to initiate contact. You can save time by > using a service like Upwork to send emails to candidates on your behalf, but > make sure that all emails read as though they come from you personally.

Please, _please_ , do not do things like this.

The rest of the advice in the doc seems okay, but it really only scratches the surface of what's involved in starting a company. One thing that seems notably absent is the difficulty of the ups and downs in a startup, and the challenge of getting through the dark times.

<comment><USER4> To add to this I would like to say that if you do reach out to candidates be cognizant of the fact that you interrupted their life with this solicitation so be sure you provide all relevant information about the job, position, duties and details about your startup in the email so the candidate can make a decision if they are interested or not from your email.

If they have to go out and spend 30 minutes to an hour to google stuff about you then it's probably not going to happen.

Lastly if they don't respond to a LinkedIn message or email within a week just move on and don't keep sending follow up emails. There is a reason the candidates didn't respond because they don't want to and shouldn't have to spend even 5 minutes responding to your unsolicited email

Thanks

<comment><USER5> This is by far the most important aspect to me with getting a message. I've only been sent around 2-4 messages with this much detail. These are the ones I actually reply to, even if I'm not interested at the moment. It's usually somebody who I'd at least want to stay in touch with. And often, if they've written something like that, it's usually something I'm somewhat interested in.

Maybe the cost versus the returns on these kind of messages aren't worth it to companies. But I really appreciate them.

</comment></comment></comment><comment><USER6> I find these recommendations useful, and more importantly, I think it shows how the authors of the guide (who are partners at V1) think about building scalable startups. I wish all funds were this transparent!

Anyone considering taking investments from V1 should read this guide because it shows that:

1\. They will point founders to the best external advice the team has come across (including most of the solid references pointed out by ignoramous below). This means they won't come up with a bunch of BS that is not best practice.

2\. That the V1 team cares about its founders! If you see the section "Invest in yourself", it's clear that they care about YOU as a founder! I love this human approach - that's NOT the case for every investor out there!

Disclaimer: Version One Ventures is 1 of 33 investors on our captable - <USER2> they stand (very positively) out!

</comment><comment><USER7> So there's gotta be a million of these. What's the best one for pre seed to post seed, startup school and paul graham?

<comment><USER8> Startupschool lectures are really what you're looking for [0]. Paul Graham's essays are worth the time too [1]: He's written many classics in a well structured persuasive style. Sam Altman's _Startup Playbook_ [2] is super nice as is Marc Andreseen's _Startup Guide_ [3]. Some VCs like A16Z [4] and FirstRound Capital [5] post great content. Stripe [6], Indiehackers [7], and Patrick McKenzie [8] are other sources that are quite not as focused on startups but worth a look nonetheless.

Twitter is a gold mine where you'd find relevant content, but you'd have to literally _mine_ it out.

I'd say though you could drain yourself going through all of this, but the fact remains that every company is different and what works for someone might not work for you. Also, there's lot of contradictory advice you need to sift through, a lot of obvious advice you need to internalize, a lot of advice you should simply ignore, and a lot of counter-intuitive advice you need to accept [9].

\---

[0] [https://startupschool.org/library](https://startupschool.org/library)

[1] [http://paulgraham.com/articles.html](http://paulgraham.com/articles.html)

[2] [https://playbook.samaltman.com/](https://playbook.samaltman.com/)

[3] [https://pmarchive.com/guide_to_startups_part1.html](https://pmarchive.com/guide_to_startups_part1.html)

[4] [https://a16z.com/content/](https://a16z.com/content/)

[5] [https://firstround.com/review/](https://firstround.com/review/)

[6] [https://stripe.com/en-us/atlas/guides](https://stripe.com/en- us/atlas/guides)

[7] [https://www.indiehackers.com/learn](https://www.indiehackers.com/learn)

[8] [https://www.kalzumeus.com/greatest- hits/](https://www.kalzumeus.com/greatest-hits/)

[9] [https://blog.ycombinator.com/advice-for-first-time- founders/](https://blog.ycombinator.com/advice-for-first-time-founders/)

<comment><USER9> > Twitter is a gold mine where you'd find relevant content, but you'd have to > literally mine it out.

This is true and by comparison, Facebook isn't. It's a barren wasteland.

</comment></comment></comment><comment><USER10> Interview homework & backchannel reference checks. Sounds like a complete nightmare.

</comment><comment><USER11> 1\. Click on PDF.

2\. Cmd-F

3\. Enter _profit_ , hit return.

4\. Zero results.

5\. Close PDF.

<comment><USER12> Handbook on how to build an army, that is going to be slaughtered 99% of the time.

</comment></comment><comment><USER13> I feel like a handbook for startups shouldn't start with a guide on how to hire.

<comment><USER14> As we mention in the intro to the guide, we don't address product-market-fit questions in the handbook (or other things you need to figure out in the earliest stages of your company creation) - we are really focused on the time between seed and Series B and hiring is a key aspect of that phase.

</comment></comment><comment><USER15> Heads up: your ePub version still embeds a title "A Guide to Marketplaces" (can be seen in Apple Books app) and the ePub content is pre-formatted and does not reflow on small screens.

<comment><USER16> Also from Opera and Google Chrome Mobile I can't download the pdf version... I just get a corrupted file.bin from the Dropbox :/

<comment><USER17> Same, here is a fixed version: [https://drive.google.com/file/d/1-f7GSHhGWJ2YfgJBf1xxJ43vuZL...](https://drive.google.com/file/d/1-f7GSHhGWJ2YfgJBf1xxJ43vuZLUcUl9/view)

<comment><USER18> Thank you!</comment></comment></comment></comment>


Yahoo looking to acquire Tumblr? - <AUTHOR> http://news.cnet.com/8301-1023_3-57584967-93/yahoo-tumblr-tie-up-in-the-works/ <comment><USER2> As a pretty heavy Tumblr user (I run a news site there with nearly 100k followers) I'm extremely bummed-out by this news, partly because it seems like a terrible fit and could hurt the dynamic of the product significantly.

On the other hand, Tumblr has been bad about monetizing its product in a way that benefits both its users and itself. (David Karp is on the record as saying that he doesn't want to impose a revenue model on users—something he says would hurt creativity—most recently at GigaOm's paidContent Live conference.) I could see Yahoo taking over Tumblr and being more flexible on this front, which could be a good thing for Tumblr creators, many of which have been forced to troll for traffic on other networks despite the fact that their base is on the Tumblr dashboard.

Having attended that conference, it didn't seem clear to me that Karp had a lot of answers as to how to improve the monetization situation for both the company and end users (they've been pushing a blanket ad model that isn't targeted but is focused on exposure-style marketing), so it could be good on that front.

But on the other hand, this community already has a lot of frustrations with the way Tumblr is operated, and a purchase like this could scare much of that audience away.

I'm sure an exit at this point makes sense for the company. But the whole thing sort of bugs me.

<comment><USER3> Would you pay for tumblr / any premium features?

<comment><USER2> Yes. Among them: Analytics that gave me some granular details on the path of a reblog. Ways to promote content that didn't annoy users (i.e. pinned posts). Security services to protect my site (two-step authentication, the ability to use something like Cloudflare). And group functionality on primary accounts, something Tumblr has never offered despite the fact that users have long asked for it.

I'd pay $20 a month for extra features, considering that's still less than what it'd cost me to host a similarly-sized Wordpress site and not have it crash on me during a traffic surge.

</comment></comment></comment><comment><USER4> This would be a really great idea for Yahoo. Their business model is selling ads on content, and the growth of mobile is leaving them behind. Tumblr generates enormous volumes of content and is making the majority of its revenue off mobile advertising already. In addition, Yahoo's demographics skew old, while Tumblr's are much younger.

It's less clear why it would be a good idea for Tumblr, other than being a really great exit for their investors and early staff -- but at the right price, that could be enough, which is probably why the number we're hearing is so large.

The biggest risk for both parties is Yahoo somehow killing Tumblr by starving it of resources (like they did with Flickr, and arguably with Delicious). I think Marissa Mayer is smart enough to learn from that mistake -- she has been pouring resources into Flickr, although it's probably too late to save it.

(Disclosure: I am a former Yahoo)

<comment><USER5> _In addition, Yahoo's demographics skew old, while Tumblr's are much younger._

The "younger" part of that equation isn't a good thing. If Yahoo makes _any_ changes to Tumblr, the userbase will respond violently.

<comment><USER4> I don't see why Yahoo would make any visible changes to Tumblr[1]. They care about the ad inventory; if they are smart they will leave the actual form of the site up to the people who invented it.

[1] although, as with Flickr, any changes that people don't like will thenceforth be blamed on Yahoo, whether or not it had anything to do with them

<comment><USER6> If they acquire the company, they'll need to monetize it which is a significant visible change. Tumblr's core demographic is notorious for responding to advertising very poorly, and the founder is clearly disinterested and clueless when it comes to monetization.

<comment><USER7> This is a dated sentiment. According to this Forbes article Tumblr is on track to make $100 million in revenue this year.

[http://www.forbes.com/sites/jeffbercovici/2013/01/02/tumblr-...](http://www.forbes.com/sites/jeffbercovici/2013/01/02/tumblr- david-karps-800-million-art-project/)

Tumblr has actually managed to pull off ads very, very tastefully. With brand and product recognition and well thought out ad-campaigns that feel like a part of Tumblr rather than banner ads.

</comment></comment></comment></comment></comment><comment><USER8> _"One of our challenges is we have had an aging demographic," said CFO Ken Goldman. "Part of it is going to be just visibility again in making ourselves cool, which we got away from for a couple of years."_

Finding out that a company's management thinks "cool" is something you can buy strikes me as an excellent signal for shareholders that it's time to sell.

<comment><USER9> All billion dollar companies, no matter how creative, degenerate into the "grow through acquisition" model. I just find it funny how well Yahoo under Mayer fits this paradigm, while Google (IMO) masks it better.

<comment><USER10> Mayer is just beginning to try to fix the company.

<comment><USER9> What will her Act II look like? So far, we've seen a few questionable acquisitions and some fluff policy changes. I don't get it.

<comment><USER11> Not all of her actions are going to be broadcast throughout the tech blogosphere.

</comment></comment></comment></comment></comment><comment><USER12> How many of these sorts of acquisitions by yahoo have resulted in anything other than destruction of the value of the service?

Geocities was the tumblr of its day.

Buying a shiny popular thing without reasonable and specific plans, getting bored with it, and then shutting it down or just letting it languish is not a good use of capital.

<comment><USER13> I think it's good that Yahoo's trying to diversify. The problem, is that Yahoo! is like that destructive little girl w/braces in 'Finding Nemo', benign neglect (Flickr) to worse (Delicious) typically await the quality properties they acquire. Can't they fixate on something else instead of sites I care about, please?

Marissa, there's a wonderful property called Experts Exchange - it gets a very high buy recommendation from me. There's also Ask.com - both great websites and perfect complements to Yahoo's growing stable of excellence..

Just please, leave Tumblr alone. Let it live.

</comment></comment><comment><USER14> Lots of expensive acquisitions.

Was Mayer known for being good at capital allocation/m&a at Google? I've got the feeling this isn't going to end well.

<comment><USER15> I agree that I don't think it will end well but their stock price has been going up so these expensive acquisitions are paying off, at least in the short run. It's also making these "expensive" acquisitions, not so expensive since they have more capital to use.

They _must_ integrate these assets quickly and do it well or it'll be over for Yahoo. Historically, successful acquisitions on a large scale have not been easy to do.

<comment><USER16> Why would the increase of the stock price increase Yahoo's available capital? Are they making acquisitions via stock swaps? Is their cost of capital decreasing?

<comment><USER11>
      | Are they making acquisitions via stock swaps?

That's one way. There are ways for Yahoo! to create more stocks and sell them, or sell stocks that are currently held by the company.

</comment></comment></comment></comment><comment><USER17> I am thoroughly unimpressed by Marissa Mayer at this point. Yahoo is doing nothing but keeping up their 'trend' of buying too early or too late.

Tumblr while a great asset is going to be FAR to expensive to pay off for Yahoo at this point, and a lot of the junior assets they have purchased have no or little direction.

In my opinion Yahoo is on a quick slide towards 'AOL territory' where they have to slim down to their 'core' to make any money and will likely suffer innovation starvation until they are almost totally irrelevant.

<comment><USER18> If you owned Yahoo stock, you might be more impressed. The price has risen nicely since Marissa joined.

Check out their weather app for mobile. It has high-resolution photos of your city in relevant weather.

[https://itunes.apple.com/us/app/yahoo!-weather/id628677149?m...](https://itunes.apple.com/us/app/yahoo!-weather/id628677149?mt=8)

</comment></comment><comment><USER19> Well, just about everything about Tumblr seems crufty, unusable, and poorly thought out to me, so I guess having it murdered by a Yahoo acquisition wouldn't be all bad.

<comment><USER20> Care to elaborate? I love Tumblr's approach (though not necessarily the content it attracts) to blogging.

<comment><USER21> I'm assuming that _hamburglar_ is talking about the backend that Tumblr relies on, and it's not pretty. They had big troubles with stability at one point last year, including a typo in VIM on a production server that spat out all of their API keys to every user.

[https://gist.github.com/zachinglis/29c5c5970d1f3313abd1/raw/...](https://gist.github.com/zachinglis/29c5c5970d1f3313abd1/raw/9fdb9a15feee6528a43ea3c7b16dc5423b01f92b/gistfile1.txt)

<comment><USER22> I've seen that leaked PHP code before... does anyone have a link to the full story to go with it? Was it really only last year?

The code is scary in so many ways. I assumed it was from Tumblr's early days and they long since fixed it up.

<comment><USER21> I'm not sure if there is any official story, but it is clearly somebody messing up with `vim` on the production servers. They've tried to go into insert mode, failed, and not realised their mistake until after they've written the file. If I remember correctly, this output was the response to every tumblr request for about 15 minutes before anybody noticed and fixed it.

        i?php

</comment></comment></comment></comment></comment><comment><USER23> In a world where Instagram is worth $1B, Tumblr would be a bargain at that price. It fits nicely into Yahoo's content strategy. I like it.

<comment><USER24> On the contrary, Instagram was probably a much more valuable company from a pure user growth + engagement perspective.

<comment><USER25> Unfortunately it appears that Facebook was prepared to pay almost anything to get it.

</comment></comment></comment><comment><USER26> If the goal is to acquire talent, and yet, keep Tumblr running, how is that talent benefiting Yahoo's current offerings?

Add to that the relief of financial exist for most of the stakeholders, how can you possibly keep motivated people around? This is almost reckless spending for Yahoo! shareholders.

wtf?

<comment><USER27> It can't possibly be for talent. As an armchair advisor I'd have to guess that Yahoo might be looking to buy its own social network to compete with G+ and Facebook.

Maybe someone at Yahoo has figured out a way to monetize Tumblr or something like Tumblr.

</comment></comment><comment><USER28> I'm very curious to know more about the terms for all these recent acquisitions. If I were Marissa Mayer, I would buy up a ton of the Series A crunched startups and bring them all together under the same roof, segregated as much as possible from any business unit that doesn't reflect the Google- esque culture she is trying to build. Eventually, once she has enough fresh talented bodies from acquisition, she could start moving the locus of power and business units over to these fresh talented acquihires from the old- timers.

I could totally see this being a convincing proposition if I were the founder of a startup that Yahoo approaches to acquire.

</comment><comment><USER29> As I thought I will not be affected by this deal if it push through, I remembered that I'm using Tumblr for my blog. I migrated from Blogger to Tumblr because I think Tumblr is so simple. Posting a blog is just one click (I don't remeber how many for Blogger).

With this, I think the rumored $1B acquisition price is worth it. Without a good (or any) business model, I'm worried that Tumblr will eventually die out. This offer by Yahoo is a win win situation for both. I really hope that Yahoo will not destroy the Tumblr feel and environment.

I'm really excited about this. Any what will it turn out to be. And if in any case Yahoo eventually destroys or break Tumblr, I think there will be a new opening in the micro blogging space for young people to use.

</comment><comment><USER30> If Yahoo! wanted to buy a kind of useless service to get younger users, SnapChat would be high on my list, not Tumblr. (Vine too, but I guess Twitter already got them). I'd make a bunch of $20-50mm buys, not one $1b buy.

</comment><comment><USER31> _"Of course, that was back in 1999 when irrational enthusiasm enveloped the tech industry during the dawn of the internet commerce."_

....ha. I'd call the enthusiasm "enveloping" some startups these days to be pretty irrational.

</comment><comment><USER32> Yahoo sells advertising, Tumblr has users. But a $1Bn valuation is over- inflated in today's market -- Tumblr barely broke even when they started introducing ads last year.

<comment><USER33> Compared, to say, $1b for instagram? (i still dont get that)

<comment><USER34> Instagram was starting to cannibalize the facebook youth userbase. Instagram worked its way into people's minds, vocabulary, and daily habits. The acquisition was a defensive maneuver by the paranoid Kid in Chief.

<comment><USER33> Wouldn't tumblr be the logical next acquisition for Zuck then since it is doing exactly the same.

<comment><USER34> Well, nobody knew (or knows) how big instagram will get. Tumblr has already peaked. Also, tumblr is too anonymous to be of any value to facebook. Not much to exploit there. I don't think forcing Real Name™ upon every tumblr would make for a happy Internet.

</comment></comment></comment></comment></comment><comment><USER35> Where does Yahoo keep getting it's money from? I don't now anyone who uses their platform. I'm assuming it must still be fairly popular in the US.

<comment><USER36> Yahoo.com is the 4th most visited web page in the world just behind Google, Facebook and YouTube.

</comment></comment><comment><USER37> "History doesn't repeat itself, but it does rhyme."

― Mark Twain

</comment><comment><USER25> This would be great for Tumblr. What other choice do they have?

</comment><comment><USER38> Medium seems like a much nicer acquisition target.

<comment><USER39> Medium is good, but unfortunately it would not solve Yahoo's "aging demographic" problem.

Everything seems to indicate that young people prefer images and other media content to words, which is the opposite of what Medium helps and incites writers to publish.

</comment></comment><comment><USER40> Nooooooooooooo! Let Tumblr be left alone.

</comment><comment><USER41> NO PLEASE! I LOVE TUMBLR!

</comment><comment><USER42> No. Just please NO!

</comment><comment><USER43> Way off topic, but not too far off from HN's usual commentary: I really wish there was a way to report back to sites what I'm blocking with AdBlock. I mean, very specific items that I'm blocking. Because with CNet, it's always that stupid object hovering on the bottom of the page that draws my eyes away from what I'm reading and actively reduces the visible space for the article. And that 'x' button sitting on there? It's never worked for me, in Chrome, Firefox, or IE on every system I'm worked on. I can never close it.

AdBlock needs to have a "Tell [site name] what you don't like!" button.</comment>


College, as an Asset Class, is a Bubble. - <AUTHOR> http://www.investingwithoptions.com/2009/08/college-is-a-bubble/

<comment><USER2> "College" really represents two separate asset classes: education and credentials. Education leads directly to productivity. Domain-specific education (like what college provides) is not a commodity: its value is not directly market-based. An education does not become more valuable based on demand. The value depends on who has the education, their ability to apply it to the current situation, and the current state of the domain. It can also depend, to an extent, on the number of people with similar education, but this is a very slow-acting force: it can take generations to have any effect. That sort of "slowness" will tend to retard bubbling.

Credentials are a tricky subject as well. The particular credentials we're talking about have value because people trust in their correlation with education. These are much closer to a commodity, because the job market is aware of credentials. Demand for credentials can increase their value. Credentials can, therefore, bubble.

Now, if you're going to college for the education, that's fine. You'll get the same result regardless of what the market for credentials does. If you're going for credentials, you might get caught in the bubble. What's worse, if you're going for the credentials, your education _will_ be of a lower quality than someone going for the education.

<comment><USER3> The measurable benefits of college (i.e. technical abilities) are largely obtainable without going to college. A substantial investment in effort and materials is required, and perhaps the final product does not attain that achieved by a college education. But the difference in achievable technical skill probably does not warrant the difference in price between the two options (excepting cases like certain sciences where experience with specific laboratory procedures is highly valuable).

The price of college is therefore primarily the price of a credential, and of the "intangibles." As my parent argues, credentials are bubblable; and as the OP argues, the market value of intangibles is (nearly by definition) a product of social factors, and advertising - hence subject to "bubbles" (by which I take it we mean wild price fluctuations unrelated to the underlying asset's value).

Personally, I think the price of college is really the price of a credential of the intangibles - belonging to a certain class in which there are a set of prioritized attitudes and shared cultural idioms, and a shared experience. This hypothesis suggests an incentive toward uniformity of experience among educational institutions - which (I opine) can be observed quite widely.

<comment><USER2> I disagree with your first point, but I upvoted because it is a well-stated, intelligent argument.

While it's true that you can get an education without going to college, college is a more efficient mechanism than most. They typically cover a variety of things that are not obviously important to someone studying independently (algorithmic complexity would be a good example), and (much like with a startup) the possibility of failure is an effective motivator.

It's certainly not the best solution for everyone -- not everyone learns the same way -- but there are a lot of people for whom it is a good choice.

<comment><USER4> I learned yesterday that Edward Fredkin had a high-school diploma (and one year at Caltech) when he became a full professor at MIT in 1968.

The value of a sheepskin is less now than it was then. It ain't what you know.

</comment></comment></comment></comment><comment><AUTHOR> Hi, post author here. A couple notes:

1\. It seems that the comments are focusing on the value as related to the student-- that's certainly one part of the equation, but don't forget that there is value for the financing companies that give easy credit access to students, and I feel that is where the excess demand is coming from.

2\. This is a very volatile argument, especially in educated circles because people take it personally. Whenever I have this discussion it seems to boil down to anecdotes and how _my_ degree is of value. That's something to avoid.

3\. This is part one in a series and I'm currently working on the data to test the hypothesis. I will be looking at per-major opportunity cost, total amount financed versus {cost, time} and a couple others.

4\. Just because I'm calling it a Bubble does not mean I'm calling a top in either tuition rates or financing. As the recession begins to hit states harder, more budget cuts will trickle into state public education and revenue gaps will need to be filled somehow. IDK if it's through fed stimulus or spending cuts, but it possibly could lead to an increase in tuition rates as financing is still available and the students won't recognize the true monthly cost until after graduation.

</comment><comment><USER5> The difference is that your education is not transferable. You only have one education level, and it is only useful for you.

The housing bubble was driven by speculation and people owning multiple properties, some as "investments" that they flipped for a profit.

There is no equivalent for that in education.

Education is more like health care, the cost is rising fast and we need to do something about it. It is not a "bubble".

</comment><comment><USER6> While there are certainly similarities between real estate and tertiary education, the title claim is obvious and unmitigated rubbish. A bubble is only possible when prices are drive up by speculation. Where exactly is there a speculative market in college degrees?

<comment><AUTHOR> Thanks for the feedback. Let's look at your statement:

 _A bubble is only possible when prices are drive up by speculation._

And who says they aren't? There are two sides to college: the actual paper, and the underlying financing. The latter is where I think we will find the bubble.

Analagous to the housing market, the demand for college would not be as great if easy-credit financing were not available. And college loan companies would have less incentive to give out finaid if it were not backed by the federal government. _That's_ where the speculation lies, in the packaging and securitization of tuition loans.

Or it could be rubbish.

<comment><USER6> Maybe I'm not up to date on trading fads, but as far as I'm aware, the "packaging and securitization of tuition" is an exception rather than the norm. At most, it creates a theoretical _potential_ for a speculative market and thus a bubble.

But there's still a fundamental difference: a degree cannot be sold, and acquiring one is a multi-year undertaking, which puts a tight limit on the number of degrees on person can hold.

Even the securitized aspect of the housing bubble depended very much on the inflated _resale price_ of the underlying asset to yield ROI fairy tales. With tuition, there is no resale price, and basing valuation on the cost would be absurd. The only sensible base is the income gap - and for that to be driven up by speculation would require a rather different world from ours, where companies buy and sell ivy leage graduates at inflated wages _and_ with inflated severance payments to the previous employer.

Oh yes, there is a market that works like that. Only, it's not in college graduates but in professional soccer players.

</comment></comment></comment><comment><USER7> This is only true for degrees that are easy and/or economically unjustified. For the forseeable future, I don't think this applies to someone majoring in molecular biology (for example).

<comment><AUTHOR> Absolutely. I'm currently mining data by degrees to analyze the true opportunity cost for various fields. A degree in communications, for example, may not be economically justifiable in the current environment. But we can flip it around and say a doctorate in medicine may not be viable due to the cost and amount of debt on the student. It's all anecdotal until I pull up some real data.

<comment><USER8> Hope to see your results when you finish.

</comment></comment></comment><comment><USER9> >...this thought has been in the back of my mind since we started hearing about the spectacular gains from Ivy League endowments and why they aren't doling them out to at-need students.

This is a very frustrating statement. First, unless I'm missing something, endowments aren't showing "spectacular gains" as of late. Additionally, Ivy League schools are doling out financial aid to at-need students. For example, Harvard has "no expected parent contribution" for students whose parents make less than $60k per year. ([http://www.admissions.college.harvard.edu/financial_aid/hfai...](http://www.admissions.college.harvard.edu/financial_aid/hfai/index.html)). Cornell has replaced any offered loans with grants for students whose parents make less than $75k per year. (<http://cornellsun.com/node/26757>). As a hypothetical example, parents who make $50k per year are expected to contribute $5000 per year. The student is expected to contribute $2000 and have a work-study paying $1800 per year. The student takes on no loans, gets $39,000 in grants per year, and graduates debt-free. (<http://www.news.cornell.edu/stories/Jan08/finAid.html>)

<comment><AUTHOR> Well I may have mistyped, but the "since" part of the statement dates back to the summer of '07 when we started hearing about the great moves in university portfolios due to alternative asset classes like VCPE and timber. Clearly that has changed a little bit.

</comment></comment><comment><USER10> 7 - People refuse to believe that the asset class could have the possibility of a bubble

I'm not so sure that people refuse to believe. The clip from Barney Frank is interesting as it shows how the blow hards on the hill put a "its for the people" spin on their policies. Now maybe Barney Frank is actually in it for the people and simply does not understand anything about economics and credit markets. But of this I am certain: if you look at the very long list of interests that lobbied for extending more credit to homeowners, the list does not include "The lobby for irresponsible middle-class that want to buy jet skis". The bubble was created by a multitude of interests that all profited from extending credit. Its not that they didn't believe they were creating a bubble. They simply did not care. This is the same with the dot com bubble. The people committing fraud knew very well they were lying, but it was making them rich or possibly so, and this sort of fraud is hard to pin down and prosecute.

The net effect of item 7 is the same without regard to my perspective. The ability to keep something like this from happening again may matter whether it was a matter of "belief" or "greed". The greed part can be solved by separating the politicians from the money.

As for education. There are probably a lot who refuse to believe that some aspects of education are unnecessary.

</comment><comment><USER11> I think the comparison with housing is interesting, especially given the social status implications. RE: tuition vs. CPI, I think it has been pretty well documented that tuition increases have far exceeded CPI, at least over the last 20-30 years.

It would be interesting to also examine what educational institutions will have to do should they come under pricing pressure. If tuition has to go down, how can they increase revenues from other streams or cut costs?

And as Howard Lindzon alludes to in the comments, is there a social cost? Is there an important element of kids achieving quasi-independence in the shelter of a university setting apart from the educational material itself, and if so how can this be replaced?

<comment><USER12> Maybe signing up for a few years of service work helps replace this. A number of my peers made the decision to pursue a year or two of service after graduating from undergrad, where a wide range of jobs are available (working at an orphanage, teaching at an inner city school, etc). Not only does this give you some social awakening (albeit, not quite the same as party school xyz), but now you're benefiting society a bit more too.

<comment><USER13> I hope your peers aren't unaware of the class privileges they are enjoying. Getting an undergrad and then going into a low-paying service job is tough if you're poor.

</comment></comment></comment><comment><USER14> I'm glad more and more people are realizing this. If you think about it for a short while, you can use the similarity between college and other bubbles to take advantage of the situation.

<comment><USER15> How? It's hard to buy some education, wait a month, and sell it.

<comment><USER16> Actually if you assume education is information transfer it's quite easy. You can even sell it twice or more times.

</comment></comment></comment><comment><USER17> It is so annoying when some finance person decides to apply rules of finance to things they do not apply, and then pats himself on the back for being oh so clever.

No college is not a bubble, because the definition of bubble does not include things like college education. A bubble refers to things that can be re-sold. More specifically a bubble refers to things whose sale value is much higher than their intrinsic value because of irrational expectation of ever- increasing resale value.

Education cannot be resold. Once you get some education it is yours. The only value of education is its intrinsic value, there is no resale value.

So there is no college bubble unless you are referring to college stocks. Now is college education too expensive, or overvalued? Perhaps. Are university of phoenix diplomas not worth the tuition? Probably. But one can express these ideas without making nonsensical comparisons with bubbles for tradeable commodities.

Talking about bubbles, anybody know how one becomes a "professional derivatives trader"? None of those guys seem very smart and yet they seem to make a lot of money.

<comment><USER13> I would say college cannot be resold, but it is sold in the first place. I looked up the Wikipedia article for "Economic Bubble", which defined it as "trade in high volumes at prices that are considerably at variance with intrinsic values." A college education is definitely a product, even though it is completely illiquid. I would definitely agree that many or even most people are paying more for college than it is worth, though not in every field and not at every school.

</comment></comment><comment><USER18> Seems more like an arms race to me. The people who sit out might save some money, however they could also be left out in the cold when even mcdonalds starts requiring college educations (because they can).</comment>


Neato's Robot Vacuum Is A Roomba Killer - <AUTHOR> http://singularityhub.com/2010/02/09/suck-it-up-neato-is-ready-to-kick-robot-vacuum-butt/ <comment><USER2> I used to work at iRobot. Neato doesn't understand the problem.

Consumers don't care how the robot works. They just care that it does. The whole point of robotics is to not care about the problem. Who cares if there is a map?

I also don't think people can say a roomba is dumb. It is designed with few sensors, but the software under the hood is actually why it is good. It uses lisp in a subsumption architecture, compiled down to run on a dirt cheap processor. Most companies couldn't compete with this on a cost basis if they tried. Here we see more CPU and sensors thrown at the problem, which is much more expensive.

<comment><USER3> That all depends on the consumer.

I owned a Roomba and set it to run while I was at work. I would never see it operate. I just emptied it every day.

However the chaotic way in which it patrols does mean it _cannot_ replace a conventional vacuum for the roles of spot cleaning or for a touch-up just before guests arrive. It completely falls down in that role. Even using its spot cleaning mode it would miss a fair amount of obvious debris even if it was placed right on the most concentrated areas. And for touch-up before a guest arrives? You better have a conventional vacuum. Sometimes it would be unable to exit rooms even with the guidance beacons. I would circle around and never be able to get out of a conventional doorway. I'd find it dead in one of the two rooms I wanted it to do for the day, and found evidence that it didn't tackle a room at all. About once a week. And this was a 5th generation model.

To me, the Roomba was to complement your existing upright or canister vacuum and was incapable of replacing it entirely. There would be greater success for a product that could replace a conventional vacuum entirely.

<comment><USER4> my roomba detects heavy dirt and focuses on it. it also has a manual spot cleaning button.

Finally, being able to set my robot buddy cleaning while I frantically get ready for guests is a fabulous boon.

</comment></comment></comment><comment><USER5> My only criteria for getting another robot vacuum is whether it can detect and avoid a maliciously placed cat turd.

<comment><USER6> It may be easier to teach your cat not to leave cat turds lying around than to teach your robot not to run over them...

<comment><USER7> Clearly you are not a cat owner.

<comment><USER8> If they use a litter box, add some sand to it. For some strange reason, if they shit in sand they'll never shit anywhere else. Maybe it's some remote gene from back when they existed solely in deserts, but I know my two cats _repeatedly_ ruined my sandpit as a child. You've literally got to replace _all_ the sand _and_ wash the pit out if you want them to stop.

</comment></comment></comment></comment><comment><USER9> I don't think the list of 'features' on the neato are the type that the average person would understand/care about. They both clean rooms, they look fairly similar, they both know how to charge themselves, etc. etc.

I think neato needs to differentiate themselves through some other means. Roomba has the mind-share and brand name (though neato is a good name).

</comment><comment><USER10> The cats will certainly run for their lives, just like when I pull out the vacuum cleaner. I'm sure they will adapt and it's better not to be associated with the mean thing. However, does it do fringes? I heard that the Roomba does not and the oriental carpets have fringes.

</comment><comment><USER11> I just feel like these products just aren't there yet. If I showed this video to my wife the first thing she would say is that it missed 1" around the wall and it didn't do under the couch.

For someone living in the city in a small apartment this really isn't worth it yet.

</comment><comment><USER12> Cool, I look forward to checking this out.

I've got a stack of dead Roombas that have been pretty disappointing overall.

<comment><USER13> Yeah, my g/f got one about seven years ago with the same result, so it will take a long time and a better track record before another purchase.

In the meantime she found a (slightly) more intelligent vacuuming device.

</comment></comment><comment><USER14> These guys need to find a better Industrial Designer.

</comment><comment><USER15> What about cables? How does it handle that kind of obstacle?

</comment><comment><USER16> I like my roomba, it does a good job.

</comment><comment><USER17> _It's been more than seven years since iRobot introduced the world to the Roomba robotic vacuum. Seven years without fundamental innovation..._

Gee. Aren't patents awesome?

</comment><comment><USER18> boundary marker looks like a tape...

<comment><USER19> article says "magnetic tape" - should prove better than those battery killing lighthouses.</comment></comment>


How to Read "Pearls of Functional Algorithm Design" - <AUTHOR> http://www.atamo.com/blog/how-to-read-pearls-by-richard-bird-1/

<comment><AUTHOR> I'm guilty of frequently recommending this book. I think it's _wonderful_ , but I also rarely am clear about it being something of a challenge to get through. This post reflects both on that challenge and the underlying "decoder key" for beginning to scrape meaning out of the book.

So now I will highly recommend reading this post _and then_ reading _Pearls of Functional Algorithm Design_.</comment>


Noop: A testable, legacy code friendly, opinionated language for the JVM - <AUTHOR> http://code.google.com/p/noop/ <comment><USER2> Good luck - it's nice to see someone making a language that suits their beliefs and requirements.

For me the ability to express myself succintly and to have as much power/capability as possible are the most important things in a language. Capable libraries are probably the next most important thing to me (which combined are why I like C# 3.5+).

However, I can respect the ability, especially when working with a large team of unknown (or variable) capability developers, the value in having high levels of testability and language limits.

In those cases you are tackling the problems quite well - testability allows senior developers to put checks and balances in place, the language limits help facilitate that testability and keep other developers from doing anything crazy, the focus on simplified readibility helps developers read others' code quickly, and executable documentation lets developers get up to speed and interact quickly.

Because of the legacy code support (albeit a little different in its extent) I'm reminded of <http://nice.sourceforge.net/> which I think has a few cool ideas (but is no longer developed though I could be wrong).

It sounds like there is no working code yet, and as such I'm not sure if you've devised some code examples that demonstrate what you want Noop to look like (i.e. examples of where it makes Java code better) but if/when you do you should put them on the main page.

</comment><comment><USER3> Needs code examples.</comment>


Human-consensus hurricane forecasting - <AUTHOR> http://stormpulse.wordpress.com/2007/08/19/human-consensus-hurricane-forecasting/ We're letting the cat out of the bag on our top secret feature.  We'll have the feature on the site by week's end (just in time for the next thing that comes down the Caribbean pike).<p>What do you think? <comment><USER2> The weather enthusiasts behind Stormpulse.com are applying the concept of the wisdom of crowds to hurricane forecasting. How? Create an account at Stormpulse.com and the next time there's an active storm, tell them where you think the storm is going to go by filling out a slick little form. They're going to take everyone's forecasts and aggregate them to see if you (and the crowd) can accurately forecast the the movement and strengthening of tropical cyclones.</comment>