Using a single AWS account is a serious risk - <AUTHOR> https://cloudonaut.io/your-single-aws-account-is-a-serious-risk/ <comment><USER2> For Tarsnap, I have several _completely independent_ AWS accounts. This is partly because I needed privilege separation before IAM existed; but I keep this setup mainly because it's dead simple and completely avoids the risk of user error: When I'm doing development work, I don't even have access to the Tarsnap production accounts.

<comment><USER3> By using IAM roles for EC2 you can also move more of the key management to Amazon. When you attach an IAM role to an EC2 instance Amazon will provide temporary credentials through the local metadata server and automatically rotate them for you. This avoids storing and copying your credentials, and makes it easy to separate your environments.

<comment><USER2> IAM roles for EC2 terrify me. They completely break OS privilege separation, since every user has access to the keys.

<comment><USER4> The servers I have set up this way aren't really multiuser machines, but your point is a good one.

</comment></comment></comment></comment><comment><USER5> About three weeks ago I started working on a new open source project meant to interact with AWS. Coding fast and dumb, I cut and pasted my personal AWS credentials into my source code, committed it, and pushed it to Github.

The next day I got an email from Amazon, alerting me to the problem. Apparently, they scrape github looking for just that kind of stupidity. I instantly deleted the project, but it was too late.

Amazon ended up waving the nearly $3k in EC2 charges I incurred, thankfully. I'm now a zealous advocate for making sure a person never even HAS AWS credentials. Instead, make a new user without a password for each use case, and manually select the privileges that account has.

If you have a password to AWS, you shouldn't have credentials.

<comment><USER6> I posted to the AWS forum and accidentally copy/pasted my secret key. Within 24 hours $11k of charges. I called them, they wiped them. It's amazing how quickly people find and use these things.

What kills me is there is no easy way to stop _all_ the instances for an account, in a region. It took me hours to kill all the instances. They had maxed out the number of instances in every single region. Very, very annoying.

<comment><USER7> Any idea what they were doing with them? Mining bitcoins? Hosting CnC for a botnet?

<comment><USER8> It's always Bitcoin mining. The attacker spins up a bunch of GPU EC2 instances and mines as long as they can. I don't think the profit ends up being very large (those GPU instances put out a puny amount of hashing power compared to modern mining ASICs)...

<comment><USER9> ROI for BTC mining on standard AWS pricing is about -90%. So attackers get 10% free BTC on spend.

<comment><USER10> So, out of $11k, they make $1k. That's the kind of money they need to make between once and three times a month minimum depending on where they live. Doesn't Amazon notice patterns in terms of source, scripts which are uploaded and scaling profile: Who uses 2 medium instances for a year then spins off 2,000 in 20 minutes?

<comment><USER9> Certainly there's a pattern there, but it's not THAT far away from people who intermittantly scale stuff for short bursts of huge processing. False positives in those cases for people that really intended to spend $10-50k in a short time might mean a HUGE loss of revenue and/or incurred customer support and service costs.

</comment></comment></comment></comment></comment></comment></comment><comment><USER11> "If someone gets access to your AWS access credentials, you're in trouble."

I know we're not supposed to post negative comments that don't "add value" to a discussion, but the only thing that comes to mind is "really?". Your setup is as secure as you make it. How you use your API access is up to you. Putting all your eggs in one basket is not insecure. This article doesn't actually bring to light anything important. There is no risk involved so long as you pay attention to what you are doing.

Any set of credentials, if leaked, destroys security. So... don't set yourself up to leak your credentials? I mean, come on, seriously?

*edit: I unfairly used the word "incompetent". Change to a phrase about paying attention to what you are doing.

<comment><USER12> Pay attention to what you're doing, and never make a mistake. Or have anyone in your team/company make a mistake. While you're at it, please make sure you don't put any bugs in the code also.

We're humans. We're imperfect. Sometimes a safety net isn't a bad thing.

<comment><USER13> Except for the fact that you can explicitly contain the potential damage by controlling which accounts can do what. It's not like you have to have one god account with no MFA that has the keys to the city.

</comment></comment></comment><comment><USER14> //Somewhat off topic

Using any kind of AWS account (in a personal capacity) is a serious risk in my eyes.

Its much like I'm happy to take the risk of buying shares, but don't like leveraged derivatives.

I don't want to deal with something that is not my area of expertise and has very real potential to blow up in my face.

I wish Amazon allowed me to cap spending to say 200USD...that I can plan for much easier than X thousands.

<comment><USER15> If you look around (or spend a few minutes writing one), you can use a scram switch that goes off of AWS billing alerts.

<comment><USER14> >AWS billing alerts

I don't want alerts, I want a fixed amount limit. If watching this sht was my day job then sure I'd be OK with billing alerts. I'm busy fighting other fires in my day job though & often won't see mails for days. By that time Amazon might have bankrupted me.

Amazon has some decent engineers & this is not technically difficult. Its very obviously a concious decision to not provide this very useful feature as any kind of capping feature will negatively impact revenue...but its also literally the only reason why I don't have a paid AWS account for personal use.

<comment><USER15> _You 're in AWS._ Why do you think alerts mean you need to be watching for them? Alerts come from CloudWatch, to SNS, and from that can go wherever you want them to. So do that, with the tools they've already given you. And this lets you kill only what you want to kill--I mean, do you want them to dump all your S3 buckets? Or just, say, down all your EC2 nodes?

It's not about capping features being negative for revenue, it's that the customers who want it _don 't matter_; if it's not something a company dropping $20K/month is going to use, I doubt it's really worth the time of day. You are a Linode or DigitalOcean customer, and _that 's not a bad thing_, but I think that AWS is not the right place for you and I think that they know that.

<comment><USER14> >from that can go wherever you want them to

You miss my point - its not a technical problem. As I said this isn't my day job.

Sure a hacker/sysadmin/whatever could re-route the AWS warning to whatever system. For me this would be a hobby/toy though and I can't have a toy that exposes me to a potential multi-thousand bill out of the blue. Thats not a hobby - thats a gamble.

As I said in my initial post - its like leveraged derivatives. If I were a day trader living that reality then that kind of uncertain would be acceptable. I'm not a day trader though so I don't want that kind of blank cheque responsibility on something I'm not actively watching.

<comment><USER15> Complaining that they don't cater to people who don't make them even couch- cushions money and won't use the tools they have to do _exactly what they want_ is really silly.

Not all services are intended for all types of customer. Right now in their business lifecycle, you're the kind of customer AWS should fire.

<comment><USER14> >won't use the tools they have to do exactly what they want

What I want is to reduce my _legal_ liability from unlimited to something less than unlimited.

Show me how to do that and I'll concede that you're right & I'm wrong.

<comment><USER16> I believe his point is that you aren't their target customer. It's like walking into a freight distribution center and asking to buy $0.50 worth of chewing gum. Sure, somebody might accommodate you, but complaining about them not doing so doesn't make a lot of sense.

In Amazon's shoes I'm not sure I'd offer a feature like that at all, because someone might set that to be "smart", forget about it as the business grows, and then have a bunch of servers terminated and/or data destroyed right during an excitingly busy period. It would be an extraordinarily bad customer experience. It could be much better for them to do as now and then occasionally credit people who accidentally go over.

<comment><USER17> > _and /or data destroyed right during an excitingly busy period_

The idea that service providers destroy data when a cap is reached is just plain weird. Usually your access to the data is simply blocked - then you can pony up some more $$ if you want access to it.

Similarly, it's also weird that people think Amazon doesn't care about 4- and 5-figure accounts. They do. Big accounts bring in money... but so do a lot of small accounts... and there are a hell of a lot more small accounts than big accounts. At my last job our monthly bill was $1-2k, and I'd get regular contacts from the AWS account manager, along with free training days.

I think that the real reason is that AWS is generally not at capacity, and if someone does have a hostile overage event, it's quite short term, and the only money that AWS really loses is from the wattage used by the VMs. There's no extra labour or hardware on their side. Refunding overages is good PR and not all that expensive, as long as it doesn't prevent other paying customers from accessing existing resources.

<comment><USER16> > The idea that service providers destroy data when a cap is reached is just > plain weird.

This fellow is proposing an account that's capped. In AWS-land with by-the- hour servers, that would presumably mean terminating running servers. Servers that have data in RAM. Servers with volatile local disks that get recycled as soon at they're terminated. Maybe you have an idea of how to shut down all of Amazon's 40-odd services with no risk of data loss, but I'm not seeing one.

If they're not going terminate them, then either they're going to keep charging or not. If they keep charging, then it seems like it's more a billing alert plus a service interruption. If they don't charge, then it's more like what they do now, which is waive charges for mistakes, except with a service interruption.

> Similarly, it's also weird that people think Amazon doesn't care about 4- > and 5-figure accounts.

I think they do care about them. I'm just saying they won't go out of their way for the $50/month accounts if it means putting serious customers at risk.

<comment><USER17> You're mixing and matching your incredulity. On the one hand, AWS has 40 services. On the other hand, everything is terminated like an EC2 instance... Which really isn't the case.

So, to begin with, most of their services can either be powered down (with chickenfeed storage costs that they could cover as part of the deal) or are outright free. Most things that you fork over money for in AWS can be disabled with some form of networking block. Have code in Lambda? Well, it doesn't get destroyed, it just gets blocked. Have RDS databases or Elasticache? Block access, and perhaps power them down after X time (which saves them to s3, and block storage for AWS's internal use is very cheap - _retail_ s3 is 3c/GB/month). S3 itself also gets your access cut. SES and SNS just stop processing their queues. Things like VPCs and IAM are free to begin with. The costs of keeping these things running behind a block is trivial compared to the overage charges they already routinely waive.

And then we come to EC2... and the story still isn't 'must be terminated'. EC2 instances can be powered down and ELB access blocked, leaving the config all in place and the instance's drive saved to s3 (which is where AMIs and volume snapshots/unpowered instances live). Yes, you will lose data in RAM, but you just get the account holder to accept that the machines can be shut down by AWS, just like already happens with spot instances. If the client _opts in_ to capped pricing, then they can take that into account and design their system around the sudden downing of the instances.

<comment><USER16> I certainly didn't say everything is terminated like EC2, so I don't know where you're getting that from.

And it looks like we agree: any attempt on Amazon's part to create something like this would be complicated and would still not remove the risk of data loss. An service interruption is, of course, a certainty.

So as far as I can see, the feature still doesn't make much sense. It's only really useful to people who aren't doing anything in Amazon that matters.

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER18> Another risk consideration is that anyone getting unauthorized access to your AWS account can delete all your resources and all your backups (snapshots, etc), effectively putting you out of business. [1]

One solution is to backup to a separate backup-only AWS account, with super- serious access controls (MFA and password physically locked away somewhere). Set up a "write-only" link, such that backups can be added, but never removed. This way, in the worst case, your runtime infrastructure can be decimated, but your data backups would be safe.

1 - [http://arstechnica.com/security/2014/06/aws-console- breach-l...](http://arstechnica.com/security/2014/06/aws-console-breach-leads- to-demise-of-service-with-proven-backup-plan/)

<comment><USER19> I prefer this method. Having multipe aws dashboards sounds like a nightmare. I would rather use the backup account approach with MFA tied to the administrator accounts on each aws site.

<comment><USER15> Multiple AWS dashboards is actually really easy with multiple Chrome profiles (not that I go to them often, there are APIs for a reason) and tends to encourage much better, much more isolated application design. I've worked in environments with a single account and in environments with multiple accounts and I can't imagine going backwards.

</comment></comment></comment><comment><USER20> At Zalando we have an account per team, and have been releasing a bunch of tooling to help do this securely.[1] It's not completely easy (e.g., account creation at Amazon can not be automated), but the security aspects are really nice, and it lets us give teams more or less full access to just their account.

[1] [https://stups.io](https://stups.io)

</comment><comment><USER21> The MFA Condition is a huge win, and I'm surprised Amazon hasn't built this a tool to make this easier yet.

However, I question the merit of using two separate AWS accounts. While this separation of responsibility sounds nice in theory, doesn't it introduce additional maintenance burden because you now have two accounts to administer? You can't define or manage the roles in the 2nd account without credentials to do so.

<comment><USER22> Usually those other accounts have much fewer people with access=much lower risk of unauthorized access. For instance, I'm a big proponent of a backup acct that the main acct can push to but not delete from. That backup acct can have very limited and tightly controlled access. It's unfortunate RDS does you no favours in helping out with this; you pretty much have to dump your DB and push it off AWS or into another acct's S3 bucket.

IAM is just a PITA is what this boils down to. Create an IAM policy that allows users to push updates to elastic beanstalk but not touch any other resources in the account.. It's a major, major hassle. AWS has no concept of resource groups and each service has different ways of restricting access(ec2 can do it on tags, other services you kinda have to use naming schemas and wild cards in your policies). So you are often left needing to have users with a little too much access, and/or spending a LOT of time testing and crafting IAM policies..

IAM is a really good idea and powerful in many ways but unfortunately AWS's lack of consistency and UX across individual services really shows through sometimes, and with IAM in particular.

</comment></comment><comment><USER23> I think the comment about using temporary credentials is an interesting one.

Where I work, we have a federated access system that generates temporary credentials so you can access the Console without having to sign in, and it uses our work authentication mechanism to do the initial handshake.

The thing is this process is useless if you want to use it with the AWS command line client, or any other tools that rely on the use of credentials. It would be nice if AWS adopted a plugin system or something where you could plug in a "credentials provider" of some sort, and the command line client queries that for credentials each time you make a request, instead of having to stash keys in ~/.aws/credentials

</comment><comment><USER24> Is there a good primer on AWS and what it provides?

</comment><comment><USER25> Is there an easier to use UI for setting up IAM users. I get hopelessly confused every time I attempt it, and worry about losing access to services I've already set up.

<comment><USER26> I don't think so. Mine is always a painful experience of trial and error using a combination of things cobbled together from the policy generator, manual editing, copy/pasting from AWS documentation, and using the policy simulator.

And then after hours of that I can determine that IAM policies fall way short off what I needed (isolation between different product groups in the org) and that we need to use separate root accounts.

</comment></comment><comment><USER27> I like the approach that Google cloud takes with projects. Projects are independent and isolated under a single account.

</comment><comment><USER28> What's cool is that in AWS as a developer they disable the ability to even login with username and password. You can set it up such that the only access is through a site that grants federated access.

</comment><comment><USER29> Like the solution via MFA & temp credentials! Simple & elegant.

</comment><comment><USER30> You can also use ssh forwarding to log into your servers through the SSH bastion without dropping your private keys everywhere on the bastion itself, securing access even further.

</comment><comment><USER31> I seriously question any business running just on AWS. There are plenty of other services that "look like" AWS and give you complete isolation.

</comment><comment><USER32> aye aye captain obvious!</comment>


There are many upcoming HTML 5 books for this summer, which one is best? - <AUTHOR>

On the amazon page I can see more than 4 for pre-order. One of them is by Mark Pilgrim. Is there any HTML 5 book yet that is considered a bible just like Dive Into Python or The C Programming Language? Which one do you recommend?
http://www.amazon.co.uk/s/ref=nb_sb_noss?url=search-alias%3Daps&field-keywords=html+5&x=0&y=0
Thank you. <comment><USER2> I have had my eye on HTML5 for Designers. I am no designer, but this book looks interesting to read:

<http://books.alistapart.com/>

<comment><USER3> I've ordered this but am quite annoyed at the long pre-order time. I think I placed my order about a month ago and still have a bit of a wait :(

</comment></comment><comment><USER4> Mark Pilgrim's site <http://diveintohtml5.org/> is the best resource I've found on the subject. Once you have an overview of all the new features, you might just want to read some examples and start playing with them, at least this is my approach (I'm not a big fan of programming books, can you tell?).

<comment><USER5> This is probably the best reference for those who want to learn what is new in HTML5 if they already know HTML. It is not just a quick start guide. It is a very good read and explains stuff very well.</comment></comment>


Ask HN: Calculating Acceleration range from json data in Python - <AUTHOR> I am writing a script where i have some geojson data from a gps receiver of a bike with speed, timestamp and latlong. I have so far sorted the data using natsort, which gives the output in reverse order with the relevant timestamps.
Basically what i want to do here is trim a video (max 60 - 90 seconds) based on when the rider accelerates. So i need to get the range of timestamps which i can pass onto another script for processing of the video. Is there any formula out there for calculating acceleration which is relevant to my problem ? How can i get the range?
https://github.com/uber1geek/Python-GPS-Data-Analysis <comment><USER2> How simplistic an answer do you need? (apologies if this way underestimates where your question is coming from...)

Acceleration is change in speed[1] over time. Delta V over delta T. How many meters per second did you change your speed by every second?

So for a first approximation: Take three consecutive datapoints (a, b, and c). Calculate the distance and time between point a and b (from their lat/longs and timestamps). Divide the distance by the time to get average speed between a and b. Now do the same for points b and c. Now you can subtract one from the other to get "change in speed", and you can approximate the corresponding "change in time" several ways depending on the nature of your data (if you've got reliable one-point-per-second datapoints, just use 1 second as your "change in time". if your gps data isn't that consistent, I'd first try averaging the two time intervals a-b and b-c and using that.)

Now just keep doing that iteratively for every point and the two points following it to get a stream of acceleration datapoints and their relative timestamps. Turning this into Python is left as an exercise for the reader :-)

 _Edit_ : I just re-read the full question after posting this - if you've got reliable speeds for each timestamped datapoint, ignore the first half of that, all you need to to is calculate "the speed at time b minus speed at time a divided by time difference between time a and b" , then assign that as the acceleration for time b (for simplicity, or for some new average time halfway between time a and b to bet closer to "the truth"). That'll probably get you "close enough".

Note that if you're getting typical GPS data "noise", you'll probably want to track a moving average of the calculated accelerations. The multi rotor controller guys seem to mostly use "kalman filters" to clean up and make sense out of real world noisy sensor data... For your use case of algorithmicly generating timestamps in video, you quite likely wont need to get their level of precision. Try the easy way first...

[1] I'll handwave away the technicalities of the difference between speed and velocity for at least the first approximation solution here...

<comment><AUTHOR> Super, I really appreciate your time and effort in explaining this in a very detailed manner. I have an insight on what to do next however turning it into python code is going to be a big challenge as i move forward to it.

</comment></comment><comment><USER3> clickable > [https://github.com/uber1geek/Python-GPS-Data- Analysis](https://github.com/uber1geek/Python-GPS-Data-Analysis)

<comment><AUTHOR> Thanks :)</comment></comment>


The University Has No Clothes - <AUTHOR> http://nymag.com/news/features/college-education-2011-5/

<comment><USER2> According to my calculations, assuming a normal course load and no financial aid, the students in my Stanford combinatorics class were paying $129.00 an hour, each, to listen to me lecture. And I'm not senior faculty -- I'm a postdoc a couple years out of grad school. Moreover, I'm not even properly an expert on the particular subject, I had to spend some time before lectures reviewing it and learning new material to lecture on.

No question, I'm the beneficiary of all this. I make good money to carry out independent research, and my department apologizes for those semesters where I am "burdened" with six classroom hours a week of teaching.

Good or bad, the purpose of elite universities is research, not teaching. There is certainly room to criticize this, but that's the career path the senior profs aim for. Elite universities hire the best researchers, and there is a trickle-down effect -- that is what you are paying for with a Stanford education.

Worth it? Dunno. Seems implausible from the perspective of just thinking about it, but looking around, the evidence seems to be that it is.

<comment><USER3> I cannot find it online, but I remember reading an essay by an ex-faculty member in CS who eventually went to work for Sun. He thought that academia was a rip-off, training-wise, when compared to Sun. At his job in Sun, when he trains people he spends 40 hours preparing every hour of material. In academia, we spend two, perhaps three, hours per hour of training material.

I think it's probably fair to say that people overpay to be taught material by faculty who really view the lecturing as an imposition on their time.

<comment><USER2> I think it's important to remember that students are getting what they in fact paid for.

There _are_ colleges which de-emphasize research and prioritize outstanding teaching. Harvey Mudd comes to mind as an outstanding example. But for whatever reason, top research universities have much higher clout.

I think perhaps because being at a top research university is ideal for exceptionally strong students; being around exceptionally strong students is very good for above-average students; and so on.

<comment><USER4> We always called them "teaching colleges," which is a name so accurate it hurts.

<comment><USER5> Hurts why? Because research universities do not teach much?

Teaching was definitely hit and miss at the research university I attended. I was lucky that's my freshman math and CS teachers were among the best the university bad.

I assumed that it was the same elsewhere, as it was in high school. Do Harvey Mudd, et al, have excellent teachers across the whole department and school?

Also, at good non-research schools, do students have a chance to learn research? Can students go on to try a Ph.D.?

For industry-bound future engineer/inventors, is publication -quality research important anyway, or does lab/practical work suffice?

<comment><USER4> Research universities do lots of teaching. But teaching is a secondary task for most of the faculty. That we could say "teaching college" with a straight face is what hurts. It _should_ be redundant - after all, college is a place people go to _learn_ so it should follow that teaching is a focus, but it was not.

Teaching was hit-or-miss at my undergrad university as well. That may also be true at teaching colleges, but professors at teaching colleges are hired primarily to _teach_. They are expected to spend most of their time teaching or preparing for it.

I really don't have enough experience with teaching colleges to say what opportunities the students have.

</comment></comment></comment></comment></comment></comment><comment><USER6> A quotation inside the article points to the public policy issues connected to the value of college: "This isn't just a matter of harnessing people's resources more productively, Altucher insists. It's a matter of harnessing the country's resources more productively. 'Let's take a step back,' Altucher says. 'What's the other American religion? Owning a home.' For years, the government encouraged home ownership for all citizens. 'So we got more and more loans that were considered subprime, and look what that did. The idea, the religion of home ownership for all, turned into a national nightmare, a national apocalypse instead of a religion. The same thing's going to happen here.'"

Indeed, if government subsidies for college attendance (government allocations of funds to colleges, individual grants for college attendance, and below- market-rate-terms loans for college attendance) change individuals' estimate of the cost:benefit ratio of college, without increasing the actual societal value of college attendance, then current college funding policy might end up being as harmful to the national economy as previous decade housing policy was.

P.S. This article had a previous submission (no discussion that time, perhaps because of the title change):

<http://news.ycombinator.com/item?id=2522135>

</comment><comment> DanI-S I'd like to know _why_ American university courses cost so much. The UK punches above its weight in research and quality of education, yet the maximum fee universities can charge is around $14k per year. The majority charge substantially less than this. Whilst government subsidy explains some of the mismatch, it's still very significant. In addition, the cost of buying books, which seems to make up a gigantic portion of college costs in the US, is not even remotely as large.

I can think of the following theories:

\- In the absence of a "socialist" state, US colleges provide poorer students with a free/subsidized education by charging richer students more.

\- US faculty receive massively higher salaries.

\- In a country where college education is commonly used as a mark of social status, they can charge a premium and people will still pay.

\- There is limited public understanding of the drawbacks of debt, so people are more willing to incur it by paying extreme fees.

\- It's a big bloody racket.

Do any of these hold any truth?

<comment><AUTHOR> Perhaps because they are the best in the world and there is a huge demand (from across the globe) to get in? <http://www.arwu.org/ARWU2010.jsp>

<comment> DanI-S That's a valid point, and an impressive list - but perhaps they are able to be the best in the world _because_ they are able to charge so much.

<comment><USER7> That's not always true historically, though. It seems more like a relatively recent phenomenon, but US schools were still the best in the world a century ago. Perhaps it's more accurate that they are able to continue cementing their lead by charging so much.

</comment></comment></comment></comment><comment><USER8> I've been wondering lately if the time might be right for the re-emergence of "guilds" i.e. organizations that promote the mastering of a craft. Not really "unions" in that the primary motivation would be teaching and improving, not bargaining with employers (though I realize that many modern unions do both).

A software engineering "guild" would seem to me to be a way to offer structured mentorship and certification of mastery without the need to also pay for years of busywork in humanities courses that, let's face it, are never applied by the vast majority of people working in our field.

<comment><USER9> a 'programmers guild' would be awesome. imagine learning to code at 14 instead of going to some worthless high school. you would start as an apprentice, watching an experienced programmer write code, helping fix syntax errors and spelling mistakes, while learning through osmosis. as a journeyman, you'd write simple unit tests, then more advanced functional tests, and then graduate by releasing your first app into production. by age 18 you could have four years of real world coding experience, while having earned a salary (albeit a small one) from age 14.

<comment><USER10> That's how it works with many opensource projects. You get around, contribute with docfixes, learn a little, then send a patch here and there, and sooner than you realize you can be doing contract or employed work related to the tools being used by the project, or to the project itself. IMHO, there's no shortage of well-paid work for those known to be talented by their peers.

</comment></comment></comment><comment><USER11> Universities don't charge enough as evident from the fact that there are an order of magnitude more applicants than spaces at many of them.

The more they raise the prices for those willing to pay the more money they'll have to subsidize brilliant students who can't.

"normal" people should be studying practical skills. It's a product of our egalitarian mindset that we regarded the status of research institutions as everyone's right, and then used subsidies and legislation until the elite institutions were so watered down that their degrees became a joke.

You can't give away status, status will just flee and look for darker corners to hide in.

<comment><USER12> "Universities don't charge enough as evident from the fact that there are an order of magnitude more applicants than spaces at many of them."

However, people who apply to universities in this class normally apply to many of them. Granted, some of them also get into many of them, but usually not all or most.

<comment><USER13> Same as getting 100s of resumes for a job posting. Just because you only hire 1 person for the position does not necessarily mean that you hire only from the best 1 percent.

</comment></comment></comment><comment><USER4> Quoth Altucher, _"When [my daughters are] 18 years old, just hand them $200,000 to go off and have a fun time for four years? Why would I want to do that?"_

I was in undergrad at Virginia Tech from the fall of 1999 to the spring of 2003. The total cost was less than $30,000 for all four years. Of course, keeping me fed and housed all four years cost more. I find his argument disingenuous, because only the elite private schools cost that much. Of course, it would be equally disingenuous for me to imply that everyone has access to a decent state school with relatively low in-state tuition. But surely that should be a part of the discussion.

</comment><comment><USER14> These articles always focus on liberal arts and private universities, which is completely uninteresting to me. I'd like to know the historic development of public school tuition for actually useful degrees like engineering and business, which don't have the steep salary curves of doctors and lawyers.

<comment><USER4> This is my common complaint when the issue comes up. From the article, _In a typical semester, one third of the students Arum and Roksa followed for their recent book, Academically Adrift, did not take "any courses that required more than forty pages of reading per week" and half did not take "a single course that required more than twenty pages of writing."_

I had semesters that were nothing but math, physics and computer science and they fall into the above category. There was reading in those courses, but it was mainly in a textbook, and 40 pages out of a physics textbook is _a lot_ of material. Likewise, I didn't turn in any writing assignments for these classes, but I turned in problem sets and programming projects on a regular basis.

</comment></comment><comment><USER15> I like the approach that Summerhill and Sudbury students tend to follow: after finishing "high school", they work in the world, thus earning money and gaining experience, and then they attend college if they think it is right for them. They work for their tuition, and they don't party away their school life.

</comment><comment><USER16> <http://ocw.mit.edu> among others:

Education is free. Certification is expensive.

<comment><USER17> Education is always expensive, even if not in direct monetary terms. Learning anything worthwhile takes time and effort. Time and effort that you can't apply to other things - aka _opportunity costs_. I mostly read and study in my "spare time", but even that means I can't do _other_ things.

</comment></comment><comment><USER18> College is just an avenue to _doing substantive work_ \-- which is when the real learning happens.

<comment><USER19> So those who do not attend college don't have access to do substantive work? I agree. Too many job postings minimum requirements include a college degree even if the actual work -- substantive or not -- do not need a college education. So, a person cannot start in industry in a less substantive position and move into a position which includes more substantive work.

<comment><USER18> _So those who do not attend college don't have access to do substantive work? I agree._

Wow, is putting words into another's mouth a standard HN move now?

Getting into college is not an insurmountable barrier. Find a professor who is genuinely interested in the work you're interested in. Start learning the background information required to do the work. Between the two of you, you should be able to figure out a way to work together. This works for both financial and admissions barriers.

There are also many kinds of substantive work one can do without a college degree.

</comment></comment></comment><comment><USER20> Honestly I've learned more playing around on my own than I did at university. Lectures really aren't for everyone, it's a really old-fashioned, outdated learning style. Whilst secondary schools have progressed hugely, university's have stayed with the sit their and suck up knowledge approach. Which sucks for me, because I learn by doing things. I guess I learned a lot from labs, and doing projects, which were awesome. But sitting there in a lecture just felt so removed from what I was actually supposed to be learning about. Combine that with ADHD and I don't think I'll be graduating in the next few months :( A lot is my fault, but I really can't stand lectures, unless the lecturer is exceptionally engaging.

</comment><comment><USER21> I have no data to back this up, but perhaps the reason college tuition has increased tenfold in the past 30 years is that the government (especially US state governments) is subsidizing it less and less... I work for a university (two different ones in the past 12 years), and there is less and less money from the state going into education.

I also think we need to start looking at the $40K college education vs. the $160K college education. Unless you want to be a corporate executive, do you really need to go to Yale, Harvard, Stanford, etc? I think PG once wrote that there is no correlation between going to a very expensive school and being a successful startup founder...

<comment><USER22> While I don't have data to cite right now either, a decrease in government subsidies is more likely to push prices DOWN, all else being equal. Increasing loan disbursements will, effectively, increase the available money supply in that market (of young adults entering college); and significant increases in money supply are by themselves inflationary, i.e. tend to push prices up.

To help see why, imagine you have one apple to sell. Alice and Bill are quite hungry, and both have $5 to spend. The maximum you can get from selling your apple is $5, because that's all either has to offer.

Suppose I come along and loan $20 each to Alice and Bill. Now if one offers $5, the other has the power to outbid. This new situation certainly won't DECREASE the amount of money you'll finally get for the apple, and could in fact end up dramatically increasing your profits.

This is oversimplified, but illustrates the legitimate macroeconomic essence, I think.

Of course, we have to qualify this with "all else being equal", which the real world is too messy for. I thought total student loan payouts in the USA had been increasing on average the past few decades, but don't know for sure: can anyone verify one way or another? The best I could quickly find is the "Average Debt Over Time" chart here:

[http://projectonstudentdebt.org/files/File/Debt_Facts_and_So...](http://projectonstudentdebt.org/files/File/Debt_Facts_and_Sources.pdf)

<comment><USER22> Update: while government-sponsored student loans can be considered a form of subsidy, I realize now you were referring to something else, lr. So, my comment is kind of tangential to your point :)

</comment></comment></comment><comment> daniel-cussen > [Peter Thiel's Stanford education got him] a job trading derivatives at > Credit Suisse before he returned West to join the Internet rush.

Thiel got this job by taking a math test and getting all the answers right.

</comment><comment><USER23> While the article has it's points, Altucher complaining about $200k for a college education is stupid. For every Stanford or Harvard out there, there's a Berkeley, UW Madison, U Washington, Virginia Tech, Georgia Tech, Ann Arbor, etc.

<comment><USER24> My GT degree cost ~$60k in the early 2000s. If you started today, it'd be north of $100k.</comment></comment>


Google no more? How this eclectic startup is reinventing search - <AUTHOR> http://upstart.bizjournals.com/companies/startups/2015/12/23/google-no-more-how-this-eclectic-startup-is.html?ana=e_ubj&u=Rb1k8XcZ0zfwr5ktAbeenA0aa53664&t=1450898594&page=all <comment><USER2> SYAC: It's [https://quid.com](https://quid.com) , which does not appear to be comparable to Google at all - at least, the consumer-facing search engine aspect of it. Thanks, headline writers.</comment>


A short history of flash photography - <AUTHOR> https://www.spectator.co.uk/2017/11/a-short-history-of-flash-photography/ <comment><USER2> Reminds me of this anecdote about Salvador Dali:

> In the 1940s [Salvador Dali] was commissioned by Max Factor to create a > brand name and a bottle for their latest perfume. Dali accepted the > commission but refused to follow any of the procedures most designs would > consider useful in achieving a successful result. He refused to meet his > client. He showed no interest in developing an understanding of the product > itself, its target audience, where it was to be sold or how it was to be > marketed. ... Four weeks later, the world's press descended up the Astoria > Hotel on Fifth Avenue, flashbulbs at the ready.

> Dali arrived nearly an hour late, visibly bereft of any object resembling a > perfume bottle. As he strode towards the lectern, journalists fired > questions while flashbulbs were discharged all around him. Just before he > reached the lectern, Dali conjured a handkerchief from his pocket and bent > over to pick up a still-glowing expended flashbulb. He pressed the bulb > gently against the lectern to flatten its tip and paused to allow it to > cool. Once the bulb was cold enough to handle, Dali held it aloft and > announced, "Ladies and Gentlemen, I give you Max Factor's _Electrique_!"

Source: [https://books.google.com/books?id=PJYk8PFtZf8C&pg=PT86](https://books.google.com/books?id=PJYk8PFtZf8C&pg=PT86)

</comment><comment><USER3> If you want to curb the invasiveness of flash photography, use infrared film and infrared flashbulbs.

"The Lurking Park Voyeurs of 1970s Tokyo" (moderately NWS) \- [http://www.slate.com/blogs/behold/2013/04/02/kohei_yoskiyuki...](http://www.slate.com/blogs/behold/2013/04/02/kohei_yoskiyuki_park_uses_infrared_flash_to_document_voyeurs_watching_couples.html)

Oddly, this is something that we have lost in modern times, because xenon (electronic) strobes don't throw enough infrared to be worthwhile, and most of the infrared film stocks were actually originally designed for the military to use for spotting tanks and other camoflauged installations. You can still put an IR-pass filter over a xenon strobe and shoot with an IR-modified camera but it is not as easy nor as effective as purpose-built film+lighting.

<comment><USER4> A famous example of this technique is Weegee's photos from a movie theater. You can tell infrared film was being used by looking at the subjects' eyes.

[http://www.slate.com/blogs/behold/2015/01/19/weegee_captures...](http://www.slate.com/blogs/behold/2015/01/19/weegee_captures_new_york_moviegoers_in_the_1940s_in_photographs_from_the.html)

</comment></comment><comment><USER5> This was a good article featured a while back on HN:

[http://www.photomemorabilia.co.uk/Ilford/Flash_History.html](http://www.photomemorabilia.co.uk/Ilford/Flash_History.html)

Discussion:

[https://news.ycombinator.com/item?id=15137945](https://news.ycombinator.com/item?id=15137945)

</comment><comment><USER6> It amuses me occasionally to get out some GE #5's or M2's and the young'ns are always shocked at how much light they put out.</comment>


Ask HN: WooThemes for Display Ads? - <AUTHOR>

I am looking to do some display ads for my SaaS, and I was wondering if there is an easy theme gallery for display ads like WooThemes, but for mrecs and so forth.  Do you know of anything like this? <comment><USER2> Display ad builders are generally pretty bad, imho. I've used pretty much every one under the sun and I think you're better served by using something like 99designs to get banners made.

Generally, you'll get a better overall banner from something like that. However, the Yahoo display network banner builder does have one great thing: it shows the CTR for an ad template. That can help identify templates that are winners, but take that number with a grain of salt as well.

</comment><comment><USER3> I have not used them yet but I've had several people recommend <http://www.elegantbanners.com/>. Their sample work seems nice and banners start at $20, so it may be worth a gamble.

<comment><USER2> Very interesting - good prices and all, still have to dig in to their work. One thing to always ask is not if they have creative references...ask if they have conversion rate references.

Find out if their clients will go on the record for improvements in conversion rate or how well they work for them. With banners, as always, your goal is to make money. Gauge their work based on that if possible.

</comment></comment><comment><AUTHOR> Also, Yahoo! and Google display ad builders have really ugly examples =/

</comment><comment><USER4> If you want some banners, I'd recommend: <http://bannersmall.com/> \- a lot of well known companies such as 1&1 and ELance have used them plus, their prices are cheap too.</comment>


Farming in the Sky - <AUTHOR> http://www.theatlantic.com/technology/archive/2015/05/farming-in-the-sky/392045/?single_page=true <comment><USER2> The technology has been around since the seventies when a whole wave of stories were published about how the future of farming would be in skyscrapers. I am reminded of pg's classic essay, suits are back.

[http://www.paulgraham.com/submarine.html](http://www.paulgraham.com/submarine.html)

Historically these greenhouses depended on being able to sell a higher quality product during winter months than produce imported from California, Florida and Mexico. The technology is better and now that more people are interested in organic there's an audience that will pay higher prices than the supermarket year around. But it is a small audience, these greenhouses will never feed large numbers of people.

<comment><USER3> That's a great article. I've read most of pg articles but missed this one. Thanks for sharing it.

</comment></comment><comment><USER4> There's a TON of productivity gains available using current land. India produces 5x less food per acre than the U.S. We can feed 9 billion using less farmland than we do now if all farms were as productive as the best US tech can provide. plus agriculture tech is progressing very quickly, another doubling of yields is expected in the next decade!

</comment><comment><USER5> The vast majority of crop land in the US is not used to produce food for humans directly, but rather for animals. (I don't have a source, but at as an agronomist and someone who grew up in agriculture, and someone who has paid careful attention when traveling I can say this is almost certainly the case or close enough to the case to be relevant for the below point...).

The key to growing more high value food (protein and fat as opposed to low nutrient density fiber, vitamins and water foodstuff like this system provides) is to remove animals from the equation. This is the terrible inefficiency in modern agriculture and solving it would result in tremendous gains.

I'm very excited about things like tank grown nutrient yeasts. There was an article a few weeks ago about a type that has been engineered to produce milk for instance. This type of thing is the future I believe and has great potential. A few off season indoor vegetables are great too. As condiments.

<comment><USER5> Here is a some more info...

[http://www.epa.gov/agriculture/ag101/cropmajor.html](http://www.epa.gov/agriculture/ag101/cropmajor.html)

Corn is the largest crop in the US. 84 million acres of it were grown in 2011 (try fitting that in a hydro system in an old factory). According to the National Corn Growers Association, 80% of corn in the US is consumed by animals.. poultry, cattle, hogs, fish etc.

Soybeans are the second largest crop with ~74 million acres grown in 2011. Although not as dramatic as corn, animals eat a significant share of soybeans as well. Over 30 million tons of soybean meal is used as livestock feed annually (very roughly 1/3 of soybean production according to my quick and dirty calculations).

The third largest crop is hay at ~56 million acres. This is exclusively used for animal feed.

These three crops together are many times the acreage of all other crops combined.

Manifestly, the big savings in water, fuel, fertilizer etc would be to remove the dependence on animals in the diet.

<comment><USER6> There's evidence to suggest that by eliminating herd animals, we're actually shooting ourselves in the foot[0].

I won't argue the benefits of increasing meat and animal product consumption here, but there is evidence to support that as well.

[0]: [http://ourworld.unu.edu/en/reversing-desertification-with- li...](http://ourworld.unu.edu/en/reversing-desertification-with-livestock)

<comment><USER5> Interesting point. Also there are many areas suitable for animal grazing that aren't good for crops.

The problem is grain fed meet and milk from an efficiency standpoint.

</comment></comment></comment></comment><comment><USER5> This is great if humanity can live on lettuce and tomatoes (both a small part of the taco filling last time I was at Taco Bell).

When they get alternatives to grains, pulses, meats and milk then real advancement in food production will being made.

<comment><USER7> To be fair, meat and milk are luxuries. Humanity can live just fine on grains and pulses alone. Lettuce... not so much.

</comment></comment><comment><USER8> So far it's always been a great idea, just not very sustainable.

I mean imagine building a home for every plant you eat. It'd be kind of crazy right? You'd need to provide many of the things a typical home has: shelter, electricity, plumbing, food supply, structures etc.

But this is really what is proposed in these types of farms. Instead of available soil, you build a large structure, pots, racks and tracks. Instead of rainwater and rainwater distribution systems (clouds) you build pipes and tubes and plumbing and install pumps to move it all around. And you provide the plants nutrients in this way. And then you build a large shelter to keep everything away from the elements that mostly grow the plants outside. And then you install artificial lighting and electricity systems to pump energy into the solar panels that we know as leaves, to generate food.

For the longest time this has been hugely expensive. And to feed 7 billion people with about 2-3 plants or whatever it is, worth each day (in cereal, greens, fruits and legumes etc), you need to build billions of little plant homes.

And that was just never very economical. Let's not forget that a hectare yields an average of about 5.000 kg of cereal. Wheat goes for about $200 per metric ton, so a hectare nets you about $1k.

Now that's 107k square footage. That's incredibly cheap. Now imagine you want to replace that with a hectare of factory or say the plant houses I described earlier. That's a huge capex investment in the tens of millions compared to the hectare of soil which is priced around $10-15k and can be leased for a few hundred. To compete on price you'd need very high production in cheap economies of scale to offset your fixed costs on building this huge greenhouse structure with infrastructure for every individual plant.

But that's also where the land is much cheaper. It's virtually impossible to run vertical farms on soil, so you need a soilless medium which isn't a problem, but it means you need plumbing to every plant. No biggie. But the same for light, you can't do a vertical farm on scale (i.e. rows and rows) and expect sunshine to reach the bottom plants who sit in the shade. So you need artificial lighting and heating, and that's very expensive compared to sunlight.

High value crops, sure it's fine. (e.g. see every indoor weed farm with artificial lighting etc). But as the future of agriculture (where staple foods worldwide are cheap cereals, legumes, corn etc), no way.

Unless you have very large advances in material science (cheap, sturdy materials to pump out large greenhouses cheaply and sustainably) and energy (cheap solar pv/thermal, wind, geothermal). And that's what we're seeing now. So it's definitely getting more appealing by the day and it's something I've been very interested in pursuing because I just adore the idea of automated farming on some levels (I love to hike in nature, but large corn fields aren't really nature to me and don't excite me more than the idea of an automated corn factory!), but I still don't see how the economics make sense.

And it's more than finance, it's touted as an environmental solution because these farms can be located near the point of consumption. But that's just one part of the story, and incredibly myopic if it's the only part of the story that's considered. For example the problem in community supported agriculture I see is that people will drive a car individually, once a week, to a small location that doesn't get economies of scale, to pick up a tiny amount of food. Because they're under the impression it's more environmentally friendly than a large shipping container moving their beans across the ocean. When in reality their food miles is a bit like saying it's environmentally more friendly drive 2 miles to a store, than to cycle 5 miles to a store for your groceries because 'it's less food miles that way'. Sustainability is a very serious issue and I'm not entirely convinced such factory-style farming is more sustainable (see inherent investments to move from soil to building plant houses). In terms of water usage, it's actually amazing to use a soilless medium, and water is one of this century's biggest challenges. But in terms of materials and energy... I'm very skeptical.

<comment><USER9> I think the biggest problem with land-based farms is that there is only so much land available for it, and that land is also highly valued for other properties (such as residential and commercial ventures).

I don't think it's unreasonable to see us get to a point where the only room we have to grow our crops is upwards (or on another planet, but that's even further out on the timeline).

<comment><USER10> > I think the biggest problem with land-based farms is that there is only so > much land available for it, and that land is also highly valued for other > properties (such as residential and commercial ventures).

I don't think you really understand the distribution of land use and urbanization in the world if you think construction of houses and shops are a substantial limiting factor in its agriculture. Your remedial assignment is to drive from Denver to Chicago.

As for growing plants on another planet... and shipping them back to Earth... if you've got that sort of an energy budget to waste shipping food around why aren't you just building nuclear-powered underground greenhouse-caves or some nonsense like that?

<comment><USER9> > Your remedial assignment is to drive from Denver to Chicago.

And yours is to drive from Boston to Miami. Or London to Glasgow. Or Osaka to Tokyo.

Cherry picking a route through the breadbasket of America as an example of typical land distribution between urban and agricultural doesn't make for a convincing argument.

> why aren't you just building nuclear-powered underground greenhouse-caves or > some nonsense like that?

Because any such location would be more valuable as a spot to pack more humans. Proximity (particularly when you start working on a planetary scale) to urban centers is worth a lot of money.

<comment><USER10> > And yours is to drive from Boston to Miami. Or London to Glasgow. Or Osaka > to Tokyo.

I haven't driven from Boston to Miami in one trip, but I've traversed that entire corridor. My grandparents once owned a few acres on a private lake near Eastford, CT. Lots of cool old dairy-farms, apple orchards, berry-picking, bird-watching, wildlife, _et cetera_. Most of I95 south of Alexandria is rather boring -- hours and hours of trees -- if you're starting by DC it's a little more fun to divert along US 301, passing through horse territory and then rolling through Fort AP Hill. More travel time, but on the northern part of the route you're more likely to pass the occasional roadside produce stands.

The southernmost bits are the ones I'm less familiar with, though those are possibly the most interesting as in Florida in particular there are places where urban development threatens the most sensitive ecosystems... but the Everglades were never known for agriculture. Agriculture does moreover threaten ecosystems across the midwest (one recent example: monarch butterflies have been suffering massive population drops in the past few years because Roundup-Ready crops have led to widespread herbicide application that kills the weeds that they'd use for food).

But agriculture itself is not under substantial pressure from urban land use at this time. Look, let's ask the US Department of Agriculture: [http://www.ers.usda.gov/publications/eib-economic- informatio...](http://www.ers.usda.gov/publications/eib-economic-information- bulletin/eib14.aspx) (2002).

Urban land, 2.6% of the nation's land area. Crop land, 19.5%. Grassland pastures/ranges: 25.9%. -[http://www.ers.usda.gov/media/249911/eib14fm_1_.pdf](http://www.ers.usda.gov/media/249911/eib14fm_1_.pdf)

"In 2002, urban land in the United States was less than 3 percent of total land area, but housed 79 percent of the U.S. population." \- [http://www.ers.usda.gov/media/250013/eib14g_1_.pdf](http://www.ers.usda.gov/media/250013/eib14g_1_.pdf)

Space for physical people has always been a nonissue next to space for their agriculture and other people-supporting land uses.

</comment></comment></comment></comment></comment><comment><USER11> "If every neighborhood had its own vertical farm, how many fewer semis would be choking up the metro area?"

In my area - Southern Ontario -- farmland is cheap and plentiful, a lot of it turning to brush or "hobby" farms (which usually means farms that don't really produce). In my area just outside of the metro area, everyone's property, with long growing seasons and excellent soil -- are in the measure of acres, and only grow decorative plants.

We recently as a family started considering the notion of mixing up life and moving somewhere else in Canada -- 100+ acre farms in PEI, NB, and Nova Scotia can be had for under $100,000. Many are now considered vacant land having been abandoned as farmsteads.

Whenever these sorts of articles talk about the "solution" to locally grown or a food crisis, I compare these two facts and something is not meshing.

<comment><USER8> In your considerations, what did you have planned to do if you did end up buying a farmstead? Pick up farming or livestock, or just live on the countryside with a big ranch available without animals but otherwise live and work like you do now presumably in a city, or a combination?

I've been really interested in the idea of moving out of the city for some time but I don't know many people who've done so. They usually just moved to a smaller city (i.e. a village) without any real farmland, but a decent backyard. (something pretty rare here in Amsterdam).

Thanks

<comment><USER11> Professionally I can essentially work anywhere with a high speed connection, and my wife is in healthcare and can work anywhere near even a somewhat built up area. We would like to do low-intensity farming, have some chickens (primarily for eggs), ideally have some woodlands, and so on. My wife really wants to try keeping bees, and the honey products from that.

I'd love to have enough land to have a small nanny-suite house, a bunkie off in the woods somewhere, etc. With connectivity (which is starting to appear in even really rural areas via wireless options), I really have to imagine that more of an exodus out of urban areas will happen by the people who want the more remote living and had only moved to urban areas out of proximity needs.

<comment><USER8> Man that sounds lovely. (I assume you do independent development, freelancing/consulting, or work remotely in dev?)

Just the other day I was checking out the blog by [http://foodcyclist.com/farm-blog/](http://foodcyclist.com/farm-blog/)

He's not a super techie, he has a few websites and it's not always super professionally structured, but check it out. He's basically a dude who got into farming later in life, first did an apprenticeship, and then started his own community supported agriculture (CSA) business.

His initial focus was chickens which you mentioned wanting to do. It's looks like it's pretty easy for him. He orders chicks online, they have enough food in their bellies to survive the trip in the mail (sounds crazy but apparently it's a normal thing). He has a heat lamp and a basic food/water installation. He designed his own pens for the older chicken for which he has a blueprint online for free, they're actually very neat. And then he has to do the butchering which is the worst part, but he has about 60 customers for his CSA who paid him upfront for the season (I think about 20 weeks), and he delivers one whole chicken to them per week, so about 50 per week, and he actually has a very high chicken price at about $25 or so. Anyway so all these people basically paid him in advance, so he starts with $30k and can make the investments he needs to. (seller discretionary cashflow is about $8k per year).

I can easily see how producing 1 chicken per week for your family, plus eggs, is a piece of cake. He's now also moving into other things, eggs, crops, hogs and vineyards and his own brewery. You can see some of his financial plans for 2015 here: [http://www.farmmarketingsolutions.com/about/income- reports/2...](http://www.farmmarketingsolutions.com/about/income- reports/2015-farm-budget/)

The cool thing is he's trying to be 100% transparent. Very interesting insight into small-scale farming. Providing for yourself is pretty easy, not trivial but very very doable. Providing for a CSA also looks like a very decent business, it's hard work but you can compete because people pay a premium for this stuff. On a larger scale I'm skeptical, selling wholesale really sucks and it just doesn't seem to be worth it unless you automate it (which seems only economical when you get economies of scale, i.e. a large scale) largely, or go big on certification and find a bio/eco/sustainability/local niche that wholesalers are seeing increased demand for themselves, too. Anyway I know that's not really your goal here but I thought I'd share the link :) All the best

<comment><USER11> That is a _fantastic_ blog, thank you very much. I love reading stuff like that.

I have the luxury, of sorts, that the farm doesn't really need to support anyone financially (the primary income still coming from `traditional' sources), and ultimately just having a selection of foods available for my own immediate and extended family would be awesome. I've done the home vegetable garden thing for years, and would really like to take it to the next level. One of the next things about a lot of crops is that minimal effort often gets you 70% to the best outcome, so while a dedicated farmer carefully tends everything to fully optimize, I find the bounty and selection of just a variety of lazily planted tomato plants incredible.</comment></comment></comment></comment></comment>


IKEA is sharing people's best IKEA hacks, and they're brilliant - <AUTHOR> https://www.fastcompany.com/90395305/ikea-is-sharing-peoples-best-ikea-hacks-and-theyre-brilliant <comment><USER2> Nearly all of these don't really look like the hacks I was imagining; this is mostly just interior decorating (like putting a lot of mirrors on a wall) and putting your shoes under a dresser.

</comment><comment><USER3> Wow. They didn't put the LackRack in. Homelabs mustn't be photogenic enough.

[https://wiki.eth0.nl/index.php/LackRack](https://wiki.eth0.nl/index.php/LackRack)

<comment><USER4> Or the Lack Enclosure for 3D printers...

[https://blog.prusaprinters.org/cheap-simple-3d-printer- enclo...](https://blog.prusaprinters.org/cheap-simple-3d-printer-enclosure/)

(Though to be fair, neither of the above suit the aesthetic Ikea like to promote)

</comment></comment><comment><USER5> A hack I found is they sell a wooden shelving unit whose shelf boards are sold at Home Depot for less than half the price, just without predrilling or hardware. (If you can buy a cheap corded drill and a hand saw, you can make just about any basic structure from DIY plans online)

<comment><USER6> Not quite the kind of hack IKEA would want to call attention to.</comment></comment>


Usage statistics and market share of IdenTrust as SSL certificate authority - dane-pgp https://w3techs.com/technologies/details/sc-identrust/all/all <comment> dane-pgp IdenTrust are the cross-signers of Let's Encrypt's intermediate certificates:

[https://letsencrypt.org/certificates/](https://letsencrypt.org/certificates/)

In the past month their market share among SSL certificate authorities grew to over 50%, but more recently they have been experiencing an unprecedented downward trend in market share.</comment>


More two-billion-pixels pictures of Everest - <AUTHOR> http://www.planetmountain.com/english/News/shownews1.lasso?l=2&keyid=40419

<comment><USER2> <http://news.ycombinator.com/item?id=4943361>

<comment><AUTHOR> Yeah, that's why I said "more", that link seems to only point to the one picture. This one has many many more including pictures from base camp.

<comment><USER2> Then obviously I'm missing something, because this, to me, just looks like blogspam. From the link you've provided I can't find any more images without going and starting to click about randomly.

Unless that one image at the bottom of the small amount of text is actually a different, stitched together image of the same place. In which case this could really do with some explanation.

So maybe I'm wrong, but I honestly can't see anything new about this, given the existence of the other post. If I _am_ wrong, feel free to provide some text to highlight why this page is better than, or even different from, the other.

Thanks.

<comment><AUTHOR> Click the green squares in the image, those take you to all the other images. I don't see a way to link to those other images from a URL nor a way to see all the available images (the first one has 4 squares, but many of those have subimages not listed on the first one.

In yesterday's link, I didn't see any of those links.

<comment><USER2> Ah! Found it! Click not _in_ the green square, but _on the edge_ of the green square. Finally, a reason to believe this is different from the other submission.

<comment><AUTHOR> Yeah. Exactly, on the edge of the square. Awful UX.</comment></comment></comment></comment></comment></comment>


Temperature and Temperament: Evidence from a Billion Tweets [pdf] - <AUTHOR> http://ei.haas.berkeley.edu/research/papers/WP265.pdf <comment><USER2> This makes sense from the perspective of "spending energy" vs "collecting energy" as probably coded in our systems. Assume in winter when it's more difficult to harvest food you want to keep your energy spending low and in summer when there is more food you can afford to spend more because you can spend more. Some rationale behind this statement could come from example from Gwern and how the immunologic system handles a cold:

"For example, in winter, we are cautious about deploying our immune resources. That's why a cold lasts much longer in winter than it does in summer. It's not because we're cold, it's because our bodies, based on deep evolutionary history reckon that it's not so safe to use our immune resources in winter, as it would be in summer. There's experimental confirmation of this in animals. Suppose a hamster is injected with bacteria which makes it sick - but in one case the hamster is on an artificial day/night cycle that suggests it's summer; in the other case it's on a cycle that suggests it's winter. If the hamster is tricked into thinking it's summer, it throws everything it has got against the infection and recovers completely. If it thinks it's winter then it just mounts a holding operation, as if it's waiting until it knows it's safe to mount a full-scale response" from: [http://www.gwern.net/Drug%20heuristics](http://www.gwern.net/Drug%20heuristics)

edit:typos

</comment><comment><USER3> Having lived in both Michigan and Texas, humidity is the largest factor in high-temperature enjoyment. In high-humidity, anything over 75F is difficult. But in Texas where the humidity is lower I can easily enjoy >85F.

<comment><USER4> Indeed. Running in Colorado sunlight at 90F is not bad. Running in Maryland shade at 90F is uncomfortable and could be dangerous.

</comment></comment><comment><USER5> That tweet density map on page 42. [https://xkcd.com/1138/](https://xkcd.com/1138/)

</comment><comment><USER6> Did the author just conclude that most people don't enjoy summer (defined as above 70F/21C, not some extreme 95F/35C craziness).

(Common sense, anyone?)

<comment><USER7> Many scientific studies confirm common sense. It's the process of turning assumptions into fact.

</comment></comment><comment><USER8> Link didn't work</comment>


Show HN: My self-published C# Programming book: Print, PDF, ePub and Mobi - <AUTHOR> http://blog.filipekberg.se/my-book/

<comment><USER2> Can you (or have you) blog about the experience? Tips and tricks? I am planning on self-publishing a book on Powershell.

<comment><USER3> He already did, 2 days ago on HN:

<http://news.ycombinator.com/item?id=5128197>

I'm not sure he really should have titled this Show HN as he already has shown it to us. Very recently.

<comment><USER4> It's just a bit of self-promotion. Compared to the last time[0], it seems pretty benign.

[0] [http://www.reddit.com/r/programming/comments/okpoh/more_read...](http://www.reddit.com/r/programming/comments/okpoh/more_readable_code_with_anonymous_functions/)

<comment><AUTHOR> Is there something wrong with showing off work that you are proud of? If something I share here gets upvotes then apparently some people find it interesting.

Not trying to upset anyone here, I just want to share my work and share my story, that's all.

<comment><USER5> Using several aliases to promote your enterprise is deceitful and yes, there is something wrong with that.

<comment><AUTHOR> Right, but I am not talking about the reddit stuff which was one year ago and I learnt from that mistake.

</comment></comment></comment></comment></comment></comment><comment><USER6> It's a good book for any .NET developer. Introduced a few things I didn't know about, well done fekberg.

</comment><comment><USER7> Great to the point book. Purchased this book around Christmas and read the first week of the year. I especially liked the Parallel chapter. One thing fekberg, have you tried purchasing the book via create space? I would have given up on step 4 of 16 if the book was on sale.

<comment><AUTHOR> Thanks, I'm glad you're enjoying the book!

Yeah I've purchased my book on CreateSpace a couple of times, I didn't find it much different from other e-stores, maybe it's because I had previously created an account?

As far as I know you go through: Buy -> Shipping address -> Shipping method -> Payment -> Receipt. Was it more steps for you than that?

</comment></comment><comment><USER8> I'd like to try it, but I can't find any sample chapter.

<comment><AUTHOR> There's a "Look inside" feature available on Amazon ( [http://www.amazon.com/C-Smorgasbord-Filip- Ekberg/dp/14681521...](http://www.amazon.com/C-Smorgasbord-Filip- Ekberg/dp/1468152106/) ). It will let you read a part of a Chapter.

</comment></comment><comment><USER9> what level of programmer is this aimed at?

<comment><AUTHOR> I've had both beginners and very advanced programmers read it and both found things that they liked.

When I wrote it I had someone that knew a bit about C# in mind and wanted to explore C# and get a gasp on when to use async/dynamic/roslyn/plinq/etc.

Check out the forewords over at <http://books.filipekberg.se> and check the "Look inside" on Amazon to get a better understanding of what you will get from the book.

<comment><USER9> I shall do that, I am a beginner starting out in C# so if I do go the route of buying it and learning from it I will let you know how it goes.

</comment></comment></comment><comment><USER10> Congrats!

<comment><AUTHOR> Thank you! It's a very special feeling when you receive the first package of proof copies.</comment></comment>


Downtime last Saturday - <AUTHOR> https://github.com/blog/1364-downtime-last-saturday <comment><USER3> This may sound selfish, but github does such a great job of writing up post mortems, that I almost look forward to their outages just because I know I'm going to learn a lot when they write their follow up.

<comment><USER4> Came here to say the same thing. This is Mark Imbriaco's wonder twin power. It really is something to generate goodwill from an outage writeup. Part of it is just unflinching transparency combined with nerdy details; you never feel like they're hiding anything, _and_ you get to learn about all the operational doodads they're working with to run at this scale.

<comment><USER5> Thanks, I really appreciate that.

For me, the motivation for transparency came from too many frustrating instances of being kept in the dark after things had gone wrong. The worst thing both during and after an outage is poor communication, so I do my best to explain as much as I can what is going on during an incident and what's happened after one is resolved.

There's a very simple formula that I follow when writing a public port-mortem:

1\. Apologize. You'd be surprised how many people don't do this , to their detriment. If you've harmed someone else because of downtime, the least you can do is apologize to them.

2\. Demonstrate understanding of the events that took place.

3\. Explain the remediation steps that you're going to take to help prevent further problems of the same type.

Just following those three very simple rules results in an incredibly effective public explanation.

<comment><USER6> This sort of approach is the reason that when I need to upgrade to a higher plan on Github, I don't flinch. In fact, I _love_ giving you guys more money, simply because you make my life completely painless; I don't think I can say the same about any other service. Keep up the awesome work.

</comment></comment></comment></comment><comment><USER7> Wow, I'm very glad our company chose a routed design with an interior routing protocol (OSPF.) I've never been able to push the limits of a layer two network as far as GitHub. A routed network helps segment the network so when systems fail or a re-convergence mistakenly occurs only a few racks are having problems and not the entire system. It's also very helpful for us to push routes to our exterior routing protocol (BGP.)

I also find it interesting they don't use any additional out-of-band network for heartbeats/management especially as unstable as their layer two network has been. It sounds like file servers need a secondary stable heartbeat network even if it is only 10/100. No judgements being passed here it just seems like a lot of eggs in one basket. That being said, Thank you for this write-up and sharing so openly and honestly. Happy GitHub customer here!

EDIT: Yes, I know routed networks can have similar problems but they are designed for routing, pathing and redundancy with a lot less overhead on the broadcast domain.

<comment><USER5> You're right on all counts. We have a great many plans with regard to how we want our network to operate that are underway.

</comment></comment><comment><USER8> Note to Github:

Freeze prod changes two weeks before and two weeks after all major holidays.

Your employees probably don't appreciate the hassle when all they are thinking about is "YEAH! DAYS OFF!"

Just my opinion and how I run my systems in the DC.

<comment><USER9> Holidays are actually one of the best times to be making changes as traffic is significantly lower, and IMO, one should be aiming for an infrastructure where you can always ship changes without being afraid of the ramifications.

Architecturally that may mean many things - hitting "SHIP IT!" might push code into a staging environment for some final testing before delivering it onto a platter in production. Should you have multiple sites, it might involve rolling out the new stuff to just one of them until you see how it goes. Maybe you have feature flags and want to introduce a new change to all servers, but just 1% of the user population?

Fundamentally hitting "SHIP IT!" should be doing just that. Any constraints you put on how fast it gets to 100% of the user population are a risk control, and you need to optimize for a balance of developer happiness and system stability.

When you concede "We can't make changes because we're frozen" outside of a critical systems ('life critical') environment, you should quit your IT job and go become a fisherman or something.

<comment><USER8> Shipping code changes is a different beast.

I am talking about the infrastructure side of things.

I have built large scale percentage-deployment, slice deployment (whatever you want to call them) scenarios like you speak of but modifying an AGG switch that provides connectivity to your entire prod space... Uh.. Go ahead and use your philosophy for managing large infrastructure and I will enjoy my days off thanks.

This change is not a SHIP IT! change. This is a switching infrastructure upgrade. This is not a push from your CI into your rolling rel. system that updates prod applications.

This is an underlying infrastructure change with high impact and high visibility with many stakeholders at risk.

Sorry for any confusion that my, very vague, post caused.

Maybe someday I will become a fisherman. But for now, I will keep these switches and servers up and running with 99.999% uptime. It is what I love to do!

Got any fishing tips?

<comment><USER9> I guess we have differing viewpoints, I see absolutely no fundamental reason why infrastructure should be treated all that much differently to code. It should be possible to fire off a test suite, to automate its deployment, etc.

I would agree that is not where most folks are at today.

I would argue the far more interesting discussion is how we develop and mature tools to get more folks there in future.

<comment><USER10> Since not everyone here is ops, if your holiday is going to be potentially impacted by a deployment, you are fully aware of that going into the deployment. We take note of people with blacked out dates (e.g. you booked your flight before we ever started talking about this), and everyone else impacted knows what's on the docket. While the issues are sudden, everyone at least has that nagging feeling that they might get a call to action.

I agree that we should be moving to automated infrastructure testing and stuff like that. To some extent, it may be possible via puppet/chef/auto tools, however, not all infrastructure is like that. Sometimes you have to go physically move stuff at your downtime window, and you can't do redundant wiring (particularly for network). I've been bitten network outages more than anything else, particularly with partial/undetected failures.

I think we're seeing a move to the "treat infrastructure as code" future, such as cluster fileservers (Netapp 8-cluster mode, or Isilon systems). You'll be able to "seamlessly" migrate data around, and virtual interfaces without impacting production. I'm looking forward to seeing how that changes ops.

</comment></comment></comment></comment></comment><comment><USER11> And High Availability isn't. Again

It seems redundancy protocols end up grappling each other more often than not

Unfortunately, there is no easy answer, and I'm sure Github employs people with lots of experience.

This makes me wonder about after several people working on problems like this it's still a challenge

<comment><USER12> You can't predict what you can't predict. This is what makes the Chaos Monkey experiment so interesting. And yes, HA is hard, and with many things it is hard with respect to latency.

The more latency you can tolerate, the easier HA becomes. At an extreme, if you can tolerate 1 minute latency than each request can come in and compute a most likely way to complete a request with the most authoritative set of actors. Few people though are willing to tolerate a commit taking 15 minutes, much less a couple of hours.

This was one of the most insightful things about NFS and the whole stateless design. By burdening the client with the state the server could be much simpler.

Once you get above a certain size the problems change becoming both easier and harder (easier in that you can disperse your data further, harder in that your confidence in agreement between copies takes longer to compute and thus increases latency). It would be interesting if Google shared their work on Spanner (they seemed to have tackled this problem at a large scale) and given Netflix's experience (Chaos Monkey's dad) it seems like Amazon still hasn't quite gotten the recipe right.

It is a deliciously thorny problem with subtle complexity and unexpected inter-dependencies.

</comment></comment><comment><USER13> It seems that every GutHub downtime I can recall was caused by automated failover.

<comment><USER14> I wonder how much downtime they've avoided through automated failover.

<comment><USER13> Yuh, I have the same question too :-). Hard to do cost/benefit analysis when you only see the costs.

<comment><USER14> In general, automated failover seems to make most small problems non-problems, but turns some small problems into big problems. It probably depends on actual numbers what makes sense for you app.

For some systems, I'd take getting rid of small outages -- I'll happily take an increased risk of a projected 15 minute loss of heart function becoming a >60 minute loss of heart function if it also eliminates what would otherwise be a bunch of 5 minute losses of heart function, since even the 5 minute interruptions would be fatal.

(Or, for a better example, revolvers vs. semi-autos. A revolver generally is more reliable, but if it goes out of timing, it's basically doomed, whereas a semi-auto can jam or pieces can break, but a monkey can clear, and a trained monkey can fix.)

<comment><USER15> Failover is meant to deal with hardware failures, which will tend to work just fine. But if the node you are failing over onto was already has 60% capacity and you add another 60% capacity during the failover, things are going to get worse.

The top-level systems probably need to be able to deal with increased latency or timeouts, and properly handle retries and throttling of traffic.

If you have some HA failover setup going but your alternate is already being used more for load balancing than for failover, problems like this will occur.

(I used to work on failover drivers for a SAN).

<comment><USER16> GitHub's failover problems have never been load-related. GitHub has pairs of fileservers where one is the master and the other's sole job is to follow along with the master and take over if it thinks the master is down, so when they do failover, it is to a node with just as much capacity as the previous master.

All the failover problems I can think of since they moved to this architecture 4 years ago have been coordination problems where something undesired happened when transitioning from one member of a pair to another. In this case, network problems lead them to a state where both members of a pair thought they were the master.

</comment></comment></comment></comment></comment></comment><comment><USER17> Ahhh... good old Heartbeat.

We used to use Heartbeat in a similar setup back in 2001. It was the worst architectural decision we ever made, and after one too many a failure (where STONITH/split-brain/etc killed the wrong machine, or both machines) we threw it out.

TL;DR: This will happen again. Guaranteed.

<comment><USER18> I was brought up to only use stonith via serial or other non-switched network connection. Running over the same network is bound to cause problems. But its not a great solution anyway.

<comment><USER9> Agreed. I completely agree, but the world isn't as kind. Providers often have funny rules about how you can cable things up in their datacenters, and as noted in #4 of where GitHub goes from here that needs to be addressed.

</comment></comment></comment><comment><USER8> I just realized that the Sys. Admin/Prod. Ops to Developer ratio here is crazy low. Everyone assumes I am talking about code changes when the article is about prod switching and network transit device changes.

MLAG or any LAG technology, LACP, bonds whatever should never impact the deployment of code. It should be invisible when it is working. Obviously it is very visible when it breaks though.

My heart goes out to the Github guys!

Sorry for the confusion everyone.

</comment><comment><USER19> I get the impression that issue wasn't network hardware but bad high availability design on fileserver side. Why do GitHub has failover network on same network hardware as primary network? But I am not surprised as I see this at lot of clients that they have failover network on separate VLAN on same network hardware. And, whenever they have network hardware issue, servers run into split brain problems.

The failover network should be totally different physically and logically from primary network. The heartbeat between file servers should be checked through both primary network and failover network. If a server can't be reached by its partner over primary network, it should be gracefully taken offline by partner through failover network.

</comment><comment><USER20> High Availability strikes again. No surprise there.

But I am mildly flabbergasted by the fact that GitHub uses STONITH. The technology is as safe as open-core nuclear reactor, and it works reliably only in very simple conditions.

<comment><USER21> STONITH is designed for critical failure so you don't end up with a split- brain situation, which is far worse than a dead node. STONITH is a good thing. The problem here is more than cluster wasn't configure to survive a catastrophic switching failure.

<comment><USER20> Yep, right - fencing solution that relies on the network to stop the service. How could that possibly fail?

</comment></comment></comment><comment><USER22> GitHub as a sysadmin/system engineer I feel your pain, I understand it completely and know the horrors of failures leading into multi-hour recovery. That said, you need to do better. This is a heavily relied upon resource for the open source community. Perhaps you didn't estimate this sort of growth but now you are here I'm sorry but the weight does fall on your shoulders. I heavily commend you guys on the service you've provided thus far and what you've done to actually pull all the varying language/library communities together. I honestly want to see this scale and serve 5 9s year round. You need to take a good hard look at the architecture of the stack and find a way to get it multi-homed.

<comment><USER23> Sure, GitHub is "heavily relied upon resource for the open source community" - but what is really the impact of their downtime?

You have to wait a little longer to do your push/pull/merge/etc?

Give them a break.

<comment><USER22> Remember that individuals and organisations pay for private repo hosting. Aren't they entitled to an adequate SLA of say 4 9s a year?

<comment><USER9> As with any purchasing decision you can read the T&Cs when making up your mind and are free to vote with your wallet if you disagree.

Another factoid is that Amazon Web Services (AWS) only offers a 99.95% SLA so where did the 99.99% for GitHub come from?

<comment><USER22> Why does it matter that AWS is 99.95%? "Cloud" based services like AWS and AppEngine have more relaxed SLAs. GitHub is not a cloud service, its a centralized version control repository. And the 4 9s, well that's just something I've always strived for when providing a service but I do believe we should all hold ourselves to it.

</comment></comment></comment></comment></comment><comment><USER24> I always love to read post-mortems like this. It's fascinating to see how a simple event can trigger a massive failure & we can learn a lot from them.

</comment><comment><USER25> tl;dr: It's really hard to get high availability systems right, and we still run the entire service as a single colo.

I can totally understand this kind of thing going wrong, but particularly given the service they provide, why not have a second colo, with a relatively recent clone of the repo, that you can route people to? Heck, you can likely even do an automatic merge once the other repo is working again...

<comment><USER26> Relatively recent clone? Sounds like that would screw customers up pretty bad if they don't realise the problem.

If they went to a second site, having synchronous commit to both sites is how it should be done, no? The extra latency on infrequent git pushes is far less an inconvenience than the possibility of grabbing the wrong code.

<comment><USER25> > Sounds like that would screw customers up pretty bad if they don't realise > the problem.

I think there'd be a variety of ways to have the system to fail until the customer made some kind of adjustment that indicated they grokked that there was a failure (like say... changing your upstream).

</comment></comment></comment><comment><USER27> I have absolutely ZERO knowledge on Enterprise Networking. But it strikes me that something as dump as Router, Switches or Network Equipment are still so unstable.

A Hypothetical questions, why not something a 8 ARM 64 bit core Linux computer as switches? Making more logic resides in software instead?

<comment><USER28> There is a move to put more of the switching and routing logic in software (google for openvswitch if you're interested) but part of the problem is that general purpose hardware doesn't stand a hope in hell of keeping up with interesting network data rates. You absolutely need to be doing a fair portion of the work in hardware, the CPUs just coordinate it.

</comment></comment><comment><USER29> It's about time we heard from them. I understand the timing of this was unfortunate (right before a holiday), but the trust alluded to in the conclusion of that post would be bolstered by faster post-mortems on major outages like this one.

<comment><USER30> I disagree. Other than for curiosity needs what's the value in a faster post mortem? Especially considering that a faster post-mortem would likely be a less accurate and less complete post-mortem? What is the difference in actionability for anyone between an update in 2 days vs an update in 4 days? I see none.

<comment><USER29> To me, a post-mortem doesn't just satisfy curiosity, it eases fears that the problem will return and informs me of future plans which may help prevent the problem, or may bring it back. It helps me form my own plan, since I'm a user of Github.

I for one spent my break hawking my email, in case further Github outages caused any of my automated deploy scripts to fail. I realize it's my own responsibility to write scripts that handle failure scenarios, but the fact is my company pays Github to host our repositories. Downtime happens, I'm understanding of that. But when it does, I want to know what's going on as soon as that information's available--especially when I'm on holiday.

I don't think a blog post written after they resolved all the issues would have been less accurate. It just would have inconvenienced whoever wrote it on a holiday.

Not the biggest deal in the world--it would take me a lot more than this slip- up to switch from Github. But IMO a service provider should get at least some information out faster than this.

<comment><USER5> I appreciate your point of view but respectfully disagree. The post-mortem would have absolutely been less accurate if we had delivered it sooner since we did not have the details about why the MLAG failover did not happen as expected until late in the evening on Christmas Eve. We've worked as quickly as possible since then to provide a full post-mortem.

<comment><USER14> IMO anything less than a billing cycle (minus a week or so) is pretty acceptable for a post-mortem, although a week is better.

</comment></comment></comment></comment></comment><comment><USER31> I've noticed MC-LAG / MLAG failing quite often in my case. Any details over the MLAG fail? Are we doing it wrong? Alternatives?

</comment><comment><USER32> <http://www.youtube.com/watch?v=c8N72t7aScY>

</comment><comment><USER33> Who's their network switch vendor? I'm not a networking expert, but boy - <USER2> sure seems like their switch vendor has screwed some things up royally. Or perhaps this is common with all complex network topologies regardless of hardware vendor?

EDIT: I would just like to say, along with others, I greatly enjoy their postmortems and I feel as though I learn something every time. Kudos to them for being forthright. I host my personal and professional projects with them and am supremely confident that my data is as safe with them as it is with anyone.

<comment><USER34> This is likely the reason the large cloud companies all do testing that _actively_ causes outages. I'm over-simplifying on purpose here: this is something that requires a lot of thought.

At first glance that seems foolish, but to quote you, "complex network topologies" are very prone to falling over badly. Since they all seem to be a one-off custom setup these days, how can you be sure it won't fall over?

Here are the testing approaches I know about:

1\. Netflix Chaos Monkey: [http://techblog.netflix.com/2012/07/chaos-monkey- released-in...](http://techblog.netflix.com/2012/07/chaos-monkey-released- into-wild.html) \- but that doesn't mean Netflix has it all together. They still have outages.

2\. Google does it. They have a team that goes around unplugging network cables and monitoring how fast the engineers can find and fix the problem. I can't dig it up but it was only a few months ago - hey, Google, your search engine can't find an article about you. :)

<comment><USER35> [http://www.wired.com/wiredenterprise/2012/10/ff-inside- googl...](http://www.wired.com/wiredenterprise/2012/10/ff-inside-google-data- center/all/) (search for DiRT).</comment></comment></comment>