Today's Outage Post Mortem - <AUTHOR> http://blog.cloudflare.com/todays-outage-post-mortem-82515 <comment><USER3> As always I'm glad to see Cloudflare post such detailed outage reports. They are one of the few providers I know of that is willing to go into such depth and that is one of the things I appreciate about them. That said, the outage that occurred was one that was indeed fully preventable. We don't exactly have as many locations as they do, but for internal resources at least, not pushing configuration changes to all devices (network included) is pretty standard practice. Basically I imagine for them a good routine to follow might be to script changes so that they are 'rolled out', something along the lines of push manual changes to a scripted 'random' router set (one in country A,B,C), wait 15 minutes and then push to the remaining router sets. That wouldn't work for all situations, such as if the entire network is seeing a DDoS or what have you, but I imagine they could adapt a routine that would prevent this particular scenario.

With all of that said, as a Cloudflare customer and also having a call with them tomorrow scheduled already over the WAF stuff, I find it a bit... frustrating that this is occurring now and such a kind of mistake.

Edit: As an aside, I wonder if the Puppet module for Junos will be extended to support route statements. That would make this kind of deployment much easier.

<comment><USER4> Although I agree, it would be rather hard to fight an attack if you didn't roll out fairly quickly...

<comment><USER5> The article, however, actually states that the expectation should be for the rule to do nothing, as the packets in question were much larger than the maximum packet size. You thereby have to examine this as some kind of rushed "let's try something, anything" reaction to a situation where an engineer didn't actually understand what was happening enough to make such a call: it is not surprising that the result was that they ended up landing squarely in "something even more confusing has now happened and everything is offline" territory.

</comment></comment></comment><comment><USER6> This is pretty impressive. Keep in mind most of the team is on the west coast so this happened at 1am on a Sunday and they put up a post mortem within hours. Obviously you would prefer it not happen at all, but that is a great response imo.

<comment><USER7> But this is not impressive:

"Someone from our operations team is monitoring our network 24/7."

"Someone" seems to indicate "1 person". Not "people are monitoring" but "someone". That's it, one person monitors the network? Like the single night guard at the warehouse?

<comment><USER8> Why would you assume this and judge the company on it, based entirely on an offhanded comment? You couldnt have given them the benefit of the doubt long enough to find another of several comments which clearly indicate there was a team of people working on the issue?

<comment><USER7> The question remains how many people actually are monitoring the network in order to call the first responders. Is it one person, two or five?

And my use of "not impressive" was in reply to someone who said "impressive" but more importantly thought it was "impressive" that they put up a post mortem within hours. That's nice but it doesn't answer the question that I had.

I stand behind my comment and re ask the question (since the info is ambiguous we have jgramhmc saying "small team who monitor things" and we have the blog post saying "Someone from our operations team is monitoring our network 24/7."

I don't think it's unreasonable (in the interest of transparency) to know exactly the structure and # bodies of who monitors the network at any given time. What is the human point of failure in the system?

I don't depend on cloudflare. But if I was running a mission critical operation and depended on them I might setup a site visit to actually get a feel of what is going on.

As an aside back when the .org registry got started one of the dns servers sat in an open unguarded office under a desk accessible by the cleaning person. I saw it when I did a site visit. And of course if you've been around long enough you know there was a time when the root dns servers sat unguarded in university offices.

<comment><USER9> You seem to be looking for "problems" where non exist. There could be 15000000 people monitoring it. It can still go down.

<comment><USER10> This.

It seems that you care more about how many people are actively waiting for something to break vs. how long their response takes. Also, (those people | that person) probably (is | are) the first responder. I really believe that one "first mate" watching the automated ship sail at night ready to triage a technical problem is better than 10 guards who will promptly fall all over themselves when the bits hit the fan.

</comment></comment></comment></comment></comment></comment><comment><USER11> > CloudFlare currently runs 23 data centers worldwide.

Shouldn't that always say - CloudFlare currently runs _in_ 23 data centers worldwide?

Or is that just how one would phrase that if you rent multiple racks or a cage in a datacenter? ...because I've seen that a bunch of times before from just about everyone.

Just curious.

<comment><USER12> The distinction is a bit arbitrary. As a customer you should care that their service is geographically distributed, not whether they own the buildings where the servers are kept.

<comment><USER7> "As a customer you should care that their service is geographically distributed"

Don't agree. If you own the data center you have more control over it. We had a case where the UPS systems in a data center had bad batteries and equipment went down because the batteries failed to kick in. Since we don't own the data center we have no realistic way to make inspections and make sure the right thing happens or that the batteries (or the generators) are cycled and maintained. We just have to trust. [1]

Now this may or may not matter with the way they have their redundancy setup. But owning a data center does give you more control over more things.

"not whether they own the buildings where the servers are kept."

Owning the data center and owning the buildings are two different things. Owning the building is owning real estate. Owning the data center is owning the security setup, backup systems etc. Two different things.

[1] So as not to contradict myself with other things I have said I should not say "trust" because you can always put some things in place to verify the right thing is happening (inspections, logs etc.) if you want. But if you own the place it's easier. If I own my home I can decide when to replace the HVAC so it doesn't fail in the middle of the summer. If I rent that's up to the landlord.

<comment><USER13> The downside is that this obligates you to do a wider variety of tasks. I'd be surprised if CloudFlare has enough profit margin to afford many redundant electrical engineers, physical network & power techs, etc. in 23 widely dispersed locations. It's obvious that the extra expense wouldn't have helped in this case but it certainly would consume a big chunk of money and management time.

Part of being a business is that you have to make tradeoffs in the real world rather than game-theoretical perfect moves. In many cases this means carefully writing contracts because you can't afford the certain expense and distraction of doing it in-house in the hope that the results might be slightly better.

</comment></comment></comment></comment><comment><USER14> I'm not very educated on this end of the spectrum - but I wonder if a process is possible where a rule or router update of some description is applied to one router only, testing the specific schema before pushing to the rest of the routers, thereby failing one router and not failing the rest? I understand the need to respond as quickly as possible - but as stated in this case this was already a manual response.

It appeals to my limited knowledge and non-existant experience that this would be a solution to the prevention of this occurring again in the future?

<comment><USER15> Don't sell yourself short: our ops team has been on our internal chat talking about how to do something exactly like this for the last hour or so. It's difficult at our scale to truly simulate traffic, but we should be able to roll rules out to just subsets of our network. That's already how we handle router OS upgrades. If a small handful of data centers had crashed, likely no one would have noticed because we've designed that fault tolerance in. This was a problem because the crashes happened system-wide. In the end, we hadn't anticipated that a simple filtering rule like this would cause such a router crash, which was a bad assumption on our part.

<comment><USER16> Similar to yRetsyM, my domain knowledge doesn't extend to this side, but if your network is undergoing a DDOS or some other form of attack, taking the time to test rules in a pre-production/test server seems to be quite dangerous. What does the ops team think about using multiple hardware vendors?

<comment><USER17> Doing the wrong fix is at least as dangerous as not handling the DDoS (as shown in this case). Based on their general network architecture, I would think a prudent thing would be a quick sanity test on a pre-production system if available, then deploy to the various colos in groups at intervals that seem appropriate given the nature of the change. If pre-production isn't available, then having the first group be one colo limits the production impact.

If they did some colos as vendor J and some colos as vendor C, I think it would be manageable, but I don't really know how much of the cross colo traffic is actually their routers talking to their routers. Homogeneity in networks makes things easier to manage, until a platform fault breaks everything at the same time. In this case, at least it was related to a change they had made and happened quickly, so it was easy to determine the cause; other platform faults may not be as easy to determine, but if only your vendor X colos fell over, at least you'd have your vendor C colos up and something to look for.

</comment></comment></comment></comment><comment><USER18> To the couldflare folks; It's refreshing to see you take responsibility, but I think you've been a bit too hard on yourselves by taking all the blame. First of all, what you hit was a unknown bug in JunOS, and Juniper is to blame for their part. Using some form of staging to slow roll-out of rule changes _might_ have saved you from a full meltdown, but when you're getting attacked, every second counts. Slow versus fast roll-out is one of those really tough balancing acts in your situation. You did a great job with it; by the time I saw the "cloudflare is down" post in the newest queue, it was already back up running again.

<comment><USER19> Is junpier to blame for the bug in their OS? Or is cloudflare to blame for not testing JunOS enough before relying on that OS?

<comment><USER15> Buck stops with us. We choose the hardware and software that runs on our network. We test and work around thousands of bugs in it. It was up to us to check range limits before applying them. While we'll never be perfect, one of the things I am most proud of with the CloudFlare team is how quickly we do learn from mistakes.

<comment><USER20> There is more to this story than meets the eye. This had to be an IPv6 fragment attack. Why weren't you already advertising rejection of such packets, at least for DNS? Why would your analysis software and procedures not already be checking for memory problems with rules that would need yo assemble all the fragments before matching? Seems like there is more to this story than meets the eye.

</comment></comment></comment></comment><comment><USER21> That was a pretty interesting writeup and I always like it when companies are totally (and quickly) upfront about negative events.

One thing that occurred to me though is that performing a hard reboot of the routers required calling people to physically access the devices and took some time to perform (as you would expect). Although I wouldn't expect it to be needed very often, I'm sort of surprised CloudFlare doesn't have out-of-band remote power cycle capabilities.

There may be some factor I'm not considering that would make that an unattractive option, but it does seem like it could cut down an already quick response time even further for any similar events in the future.

<comment><USER22> I've never seen remote power cyclers on big routers in major facilities which have on-site remote hands, even when servers all get both IPMI/LOM board cyclers and physical external cyclers. At most, the routers get a serial port connected to a serial port console server or directly to a modem, and/or an admin ethernet network.

I've seen smaller routers, CSU/DSU, etc. type devices in branch offices on cyclers, though.

I think it's mostly that the routers usually have both good OOB management and good watchdog (reboot on freeze) behavior, and that the PSUs in the bigger routers tend to exceed the per-port power limits of most of the external power cyclers.

It may be a good idea, though.

<comment><USER23> You would need another network (not just a vlan) to run this as well, if you are going to try to reach it when nothing else is working.

<comment><USER24> We just hook up a DSL modem to the OOB network or plug it straight into the OOB interface on a core router. You used to do this with actual modems but it's cheap enough to do it with DSL these days, then you're not dependent on any of your own network to access the device in case of failure.

<comment><USER25> We've been doing this with mikrotik boxes with either wifi or usb gsm modems depending on the what is available in the location.

<comment><USER22> Yeah, I've seen a lot of great options for OOB access: 1) At carrier hotels, wifi (heh) 2) Cellular modems (ideal for branch offices; a lot of datacenters have bad cell coverage inside the racks/cabinets/floor though) 3) Cross- connect (in places with free/cheap cross connects) to someone you don't use for transit. Can be mutual 4) Some facilities give you an OOB network, although this often has issues (if you buy transit from them, it's possible your outage is due to something going wrong with them, and it might take out your OOB access)

I'm looking at the Verizon Private-IP thing (an outsourced private network over Verizon's cell infrastructure) for OOB management of lots of CPE; the cost per device per month is low, and then you pay for bandwidth across all of them. Makes initial provisioning easier, plus ongoing monitoring/maintenance.

</comment></comment></comment></comment></comment></comment><comment><USER26> if you want to build a reliable system, one useful thing to do is use equipment from multiple vendors. sure it's inconvenient, but by doing this you can often de-correlate failures. especially if you want to improve someone else's reliability.

e.g., from simple things like hard drives in a raid from different vendors, to n-version programming in safety critical systems (like airplanes).

<comment><USER22> That works when the interfaces are totally standard, but edge/core routers are not like that. Cisco supports one set of protocols for talking to other Cisco products; another set for talking to everything else. The "everything else" protocols suck in a lot of ways (they're ok inter-site, but not really so great intra-site).

Same with Juniper. (there aren't really other viable options besides those two)

You could build the same site fully independently with all-Cisco on one, and all Juniper on another, and potentially get some better isolation from vendor faults, but at very high expense.

You end up with much _worse_ reliability if you have a mixed Cisco/Juniper network without a lot of additional isolation otherwise.

<comment><USER27> Re: "there aren't really any viable options..."

Total misconception. BGP, OSPF, ISIS, LISP, etc. are all non proprietary. Sure, the root cause of this particular problem is that CF is using something specific to Juniper, however router interoperability is not predicated on components like that. This example was a tool CF operationalized, and likely had little to do with their routing with the exception of it being a metric they may have influenced routes with.

People who have all Cisco or all Juniper shops namely do it from a cost perspective. Sure, there are some reasons outside of that but it's likely the big driver. The more you buy, the more you save. And the network sales realm is royally messed up to begin with. I've seen Juniper give 90% discounts on hardware just to break into a Cisco shop. But, the reality of the situation is that all of this gear is marked up well into the thousands of percent. So if you're not getting, minimally 50% then your probably not doing yourself due diligence.

<comment><USER22> "there aren't really any viable options" to juniper or cisco for core/edge routers.

There are some routing protocols which interoperate (which is how different sites on the Internet can talk to each other), but most of the protocols used for HA or management of a given set of routers, or, more importantly, most tested/debugged implementations of HA and device management, are Cisco or Juniper specific.

No big deal announcing routes to your upstream if you use Juniper and they use Cisco. Big deal if you have Cisco+Juniper and want to do HSRP (Cisco-only).

<comment><USER27> Well, no.

I've been in network engineering for 12+ years and I fundamentally disagree with a lot of what is said about "networking" and interop by many programmer- types (not casting here, but) on HN. Yes, yes, you may understand system DevOps to a point, however I'm not sure you've spent a significant amount of time studying Dijkstra's algorithm or truly have an idea of how to deploy a global IPv6 overlay. I'm also not trying to be snide here but I feel that, often times, many things that come up on HN are just fundamentally designed wrong from PHY all the way up until the devs get a hold of the rest. I've been in a very successful startup (think one of the top online backup services) wherein their network was run on commodity junk hardware. They were asking me how I'd troubleshoot this, that and the other thing - obviously with no debug (this guy said that with a grin). First and foremost, you designed it wrong - I can show you inefficiency in about 10 minutes of performance engineering that I would have designed around without thinking about those things. So, yes, I can waste time tracking down a bad NIC on your network, but if you feel that you've earned geek cred because you fired up Wireshark and parsed through a few simplistic ARP tables - you haven't impressed anyone but yourself. That's when I realized I was working with professional _developers_ , and not network architects.

Your simplistic view of FHRP is trivial at best. Maybe if you were talking about how you'd design fault tolerance into a virtual link, say an LSP, with something like BFD in your design I'd be more impressed than conversations about proprietary redundancy protocols of which most network engineers won't touch for a variety of other reasons than the big "C".

</endrant>

<comment><USER22> Virtually no network engineers (by percentage) have to do anything other than worry about what their vendor supports for a given configuration (and usually a fairly small set of configurations, too); it's much more about policy and operations.

Similarly very few developers have to solve open CS problems in writing a CRUD application (or I guess more comparably to ops, come up with a novel implementation of a complex algorithm).

This is progress, though.

<comment><USER27> "Virtually no network engineers (by percentage) have to do anything other than worry about what their vendor supports for a given configuration" - <USER2> statement puts a perspective on your thinking. And then I read your information on the services your company offers, and I realize that it's not worth having a discussion.

"<redacted> takes your security very seriously." - right. That's a statement, not information regarding the thought or implementation. There's not even a mention of technology. <sigh>

<comment><USER28> Be nice.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER29> So as far the DOS attack was very successful. It took down the site which it intended to and take down the network with lots and lots of the sites.

Hope lessons are learnt and your next generation is less prone to these attacks

<comment><USER30> If the attacker knew about the Juniper bug and thought about a way to convince the network operators to introduce the rule of death themselves, then this is a nice hack indeed. It won't be easy for CloudFare to generically protect against these types of attacks. They could either have mechanisms to revert configurations faster or a way to test new configurations on a single router.

<comment><USER31> The idea that the attacker knew about the bug is, I think, a remote but intriguing idea.

Wonder if Cloudflare need to do some tests along the lines of:

A) List up all the types of rules we usually use to mitigate these situations. B) Run those rules on a test router with wildly unusual input values, as was the case in this situation. C) Send test traffic using that wildly unexpected input to see what happens.

Basically a bit of manual fuzzing

Time-consuming and maybe not worthwhile, but it could save against another full system death.

</comment></comment></comment><comment><USER32> Rather unfortunate for the credibility of Cloudfare as a network provider, but you've got to admire them for their honesty and it'll work out better for them in the end. It's amazing how a few lines of code managed to bring down Cloudfare, they could have told us anything and nobody would have been able to question it; instead they gave us the truth and I really respect that. They didn't blame the intern, they didn't blame their hardware or make an excuse about a power outage. In terms of honesty Cloudfare seems to be leading the way regardless of their public credibility or image being tainted. Very impressive response time and resolution of the issue as well, good job Cloudfare!

</comment><comment><USER22> Wow, that's pretty fast turnaround for a post-mortem (although it looks to have been a simple problem, so easier to figure out what to write)

<comment><USER33> Our customers deserve to know what happened as quickly as possible.

</comment></comment><comment><USER34> Two things:

1\. That video of the BGP routes disappearing is awesome, and

2\. A 40 minute outage sounds bad, but consider the following timeline (based on the writeup):

> T+0: route change made, propagates

> T+10: Response team online, attempting local fixes

> T+30: Routers across 23 data centres in 14 countries hard reset and networks > coming back up.

</comment><comment><USER35> Funny that now the post mortem is down ...

<comment><USER33> Yes, ironic. Unfortunately, the CloudFlare blog is hosted on posterous and they seem to be down.

<comment><USER22> You may want to move off posterous before it goes down for good in a month, too :)

</comment></comment></comment><comment><USER36> > Even though some data centers came back online initially, they fell back > over again because all the traffic across our entire network hit them and > overloaded their resources.

I know very little of networking, but this seems to be a recurring pattern that aggravates many major outages. What surprises me is that this so often seems to be a scenario not accounted for.

<comment><USER37> You can only account for it by having more hardware and then it's possible more of your hardware will fail which puts you right back to where you started.

<comment><USER38> I don't think that's the only solution. I would be willing to bet that outside of heavy-DDoS conditions that even a tiny fraction of Cloudflare's network could handle the incoming tcp connections and deny all of them. At that point you don't have to worry about traffic collapsing anything. You can wait to bring up more equipment. You can send a tiny error page. You can let X% of requests get through and be fully served.

I bet that most of the time the domino effect happens to internet services in general it's with nodes that are _accepting_ most requests. They _allow_ themselves to be overloaded. An active HTTP session uses orders of magnitude more resources than simply denying the initial packet and forgetting about it forever.

<comment><USER10> You're vastly oversimplifying the problem here by only accounting for one class of problems.

>". I would be willing to bet that outside of heavy-DDoS conditions that even a tiny fraction of Cloudflare's network could handle the incoming tcp connections and deny all of them." depends on the attack.

>"You can send a tiny error page. You can let X% of requests get through and be fully served." Not usually that easy.

<comment><USER38> I said _outside_ of attacks.

I call BS on saying it's not easy to limit the number of served connections and RST the rest. Isn't this something every web server can do by itself it's so easy?

</comment></comment></comment></comment></comment><comment><USER39> this is the type of reason why i stopped using cloudflare. there are just too many eggs in one basket. it's as if their entire service becomes a SPOF to your infrastructure.

<comment><USER40> You could say the same thing about almost any of your service providers. Your DNS provider goes out, everything goes out. The routers at the data center with your servers go out, all your servers go out. Your CDN goes out, all of the static assets on your site go out.

There will always be potential SPOF.

<comment><USER5> While your example with the routers at your backend is truly problematic, DNS is designed with built-in redundancy and CDNs (which CloudFlare should not really count as) having a world-wide outage (as opposed to "people accessing from New York are currently having issues, as we lost one PoP") is nigh-unto unheard of... can you imagine Akamai (or CDNetworks or EdgeCast or even Amazon) saying "doh, all of our infrastructure everywhere just disappeared"?

The core problem with CloudFlare is that they seem to have a highly- centralized take on what is normally a massively-decentralized solution-space, with large numbers of value-adds they encourage customers to use without making it clear that they treat in a haphazard manner, doing very little testing before deploying pushing-the-envelope features while simultaneously having very little in-house debugging expertise to handle serious issues.

(As a concrete example of that last complaint, Cydia was crippled for an entire day due to ModMyi turning on CloudFlare's "preloader" transformation, which apparently caused many WebKit-based browsers--including both MobileSafari and Cydia--to entirely lock up; CloudFlare seemed to go the entire day without noticing, which I continue to be utterly _shocked_ by, and it was only after I told them how to fix it that they were able to acknowledge the issue.)

<http://www.saurik.com/id/14> <\- When "Dumb Pipes" Get Too Smart, an extensive analysis of this bug

</comment></comment></comment><comment><USER41> I think you should have investigated why you got ~90kb packages despite having a max pkg size of ~4kb instead of putting in that rule. :)

<comment><USER9> I was thinking the exact same thing.

</comment></comment><comment><USER42> > attack packets were between 99,971 and 99,985 bytes long.

This should raise a red flag, as it must be impossible. Ethernet NICs would just bail out on packets longer than what you've set the MTU to, and ethernet frames would just come from the next hop in most cases. And IP packets have a max length field of 16 bit.

<comment><USER43> <http://en.wikipedia.org/wiki/Jumbogram>

> An optional feature of IPv6, the jumbo payload option, allows the exchange > of packets with payloads of up to one byte less than 4 GiB

<comment><USER24> Yes but they were still seeing packets bigger than the MTU of Ethernet (or Sonet or whatever other layer 1/2 tech they're connected to the rest of the net with). It doesn't matter what higher level protocols can handle.

<comment><USER44> They could've been fragmented IPv6 packets. Or it could've been a bug in their profiler.

<comment><USER45> which is precisely why it seems like lunacy to roll out such an asinine firewall rule to _every_ router. if there was ever a time to "spot check" a change, this was it.

they didn't. and they paid the price. good on 'em for the quick and honest post-mortem. regardless, it was a dumb move.

</comment></comment></comment></comment></comment><comment><USER46> Impressive response. 30 minute outage for something most of the hosts I've worked with in the past would have been mystified about for hours. Then a quick RFO and promise of proactive SLA adjustments? Next time I need a CDN or attack mitigation I'll be talking to Cloudflare

</comment><comment><USER47> What I don't understand is why Cloudflare is making changes to their border routers in the process of protecting their customers. I am a network engineer and I love Juniper, but the reality is with any complex system, every change you make has a possibility of inducing an unexpected failure. I would think Cloudflare would have increased stability by using an architecture where the border routers have a mostly static config, and there is a set of firewalls (e.g. Juniper SRX 5800) behind them that are doing the actual filtering and changing configs in response to threats.

<comment><USER10> So now you have two pieces of gear to test changes on and another interaction where stuff could break / go weird.

I don't see how that would solve anything here.

<comment><USER47> The thing it would solve is risking all their BGP peerings going down as a result of day-to-day service operations (i.e. every time they add a filter).

</comment></comment></comment><comment><USER48> OT: I want to pitch cloudflare for our CDN needs. Can someone estimate the scale of cloudflare wrt. akamai (current provider), in terms of operations, consumers etc.?

<comment><USER45> akamai is about 100x the size and probably 200x the price.

<comment><USER5> If you are going to go for the $3k/mo CloudFlare plan, you are already nearing the ballpark of Akamai, and would do good to look at the many CDNs that sit in the middle of that scale (such as CDNetworks or EdgeCast).

</comment></comment></comment><comment><USER49> _Developing good software comes down to consistently carrying out fundamental practices (regardless of the technology)_ \- Paul M. Duvall

In this case: Development. Versioned change. Test or staging environment. Tests pass. Production.

<comment><USER31> Meanwhile your customers are getting DDOS'ed while you are faffing about.

Yes, I fully agree that for things like software and standard network maintenance the above is good. But as someone else mentioned in this thread. DDoSes that require quick resolution put you between a rock and a hard place in terms of doing things "right"

<comment><USER49> That's true. However, look at what happens to _all_ of your customers when you fail to test. If you haven't limited, or at least tested the extreme ranges of allowable input to a system automatically pushing out live configuration to all of your routers, there's nobody else to blame but yourself. Sorry. Are most people this diligent? No. Should we be? Yes.

<comment><USER31> I actually wondered about testing with extreme ranges in another comment, but this testing is done "offline" (Not on live routers and not in response to current circumstances).

However, at least how I read it, your comment was about testing rules in consistently in dev -> staging -> prod when you create one which I think is not viable in this situation since you are on a very tight deadline with immediate impact on your customers.

</comment></comment></comment></comment><comment><USER50> So what are they going to change as a consequence? It seems logical to not rely on a single router vendor anymore, or to test new rules on a staging setup at least for a very short time before pushing them to all routers.

<comment><USER22> Running Vendor J and Vendor C routers together means you get exposed to the weird bugs in either's open/interoperability code, and lose out on all the advanced features (since most of the good stuff isn't well supported in true cross-platform vendor independent fashion).

It's probably more reasonable to split your network into a few more independent sections and never do updates which affect everything, but unless you're building the space shuttle (and can accept vastly higher costs and lower performance), it's probably better to pick one hardware platform, at least now.

<comment><USER15> While we'll discuss it more at length and after a bit of sleep, my hunch is this will be closer to our approach.

<comment><USER22> The thing which annoyed me the most was losing all DNS. You really need to have the DNS servers in separate infrastructure (ASN, netblock, while anycasted) so there is never a case where both of your DNS are out for a customer domain. The "CNAME" product looks pretty kludgey.

<comment><USER51> By the same token you (the customer) should not have all your DNS eggs in one basket.

<comment><USER52> CloudFlare's CDN bits require you to give them DNS delegation of your stuff, last I looked.

<comment><USER22> There is a way around it with some of the premium accounts, but it kind of seems like a hack.

</comment></comment></comment></comment></comment></comment></comment><comment><USER53> Just wondering if the source of the large packets were from a [large] range of hosts or maybe a single host?

Ouch if a single host activity took down ~750k websites - whether deliberate and direct or not.

<comment><USER9> It's probably even more then that. CloudFlare hosts cdnjs which a ton of people use. It could have "taken down" millions of sites. And by taken down I mean rendered unusable.

</comment></comment><comment><USER54> Presumably Jupiter's Junos is closed-source, making investigation more difficult? Do they provide it to some of their bigger clients under an agreement?

</comment><comment><USER55> I got a "Oops there was a problem" page from Postereous trying to open the blog page/site...

</comment><comment><USER56> Now we need a Post Mortem on the Post Mortem, as it is now down

"Oh noes! Something went wrong."

</comment><comment><USER57> Posterous seems to be down.

</comment><comment><USER58> Yes, case number 384,449,194 of systems management causing a system problem. Also case number 439,224 of what looked like a localized problem quickly causing a huge system, e.g., all 23 data centers around the world, to crash.

They have my sympathy: So, they typed in a 'rule'. At one time I was working in 'artificial intelligence' (AI), actually 'expert systems', based on using 'rules' to implement real time management of server farms and networks. Of course, in that work, goals included 'lights out data centers', that is, don't need people walking around doing manual work but not the case of 'lights out' as in the CloudFlare outage, and very high reliability.

Looking into reliability, that is, putting into a few, broad categories the causes of outages, a category causing a large fraction of the outages was humans doing system management, or as in the words of the HAL 9000, "human error". Yup.

And the whole thing went down? Yup: One example we worked with was system management of a 'cluster'. Well, one of the computers in the cluster "went a little funny, a little funny in the head" and was throwing all its incoming work into its 'bit bucket'. So, the CPU busy metric on that computer was not very high, and the load leveling started sending nearly all the work to that one computer and, thus, into its bit bucket and, thus, effectively killed the work of the whole cluster.

As one response I decided that real time monitoring of a cluster, or any system that is supposed to be 'well balanced' via some version of 'load leveling', should include looking for 'out of balance' situations.

So, let's see: Such monitoring can have false positives (false alarms) and false negetives (missed detections). So, such monitoring is necessarily essentially a case of some statistical hypothesis testing, typically with the 'null hypothesis' that the system is healthy, applied continually in near real-time. So, for monitoring 'balancing', we will likely have to work with multi-dimensional data. Next, our chances of knowing the probability distribution of that data, even in the case of a healthy system, is from slim down to none. So we need a statistical hypothesis test that is both multi- dimensional and distribution-free.

So, CloudFlare's problems are not really new!

I went ahead and did some work, math, prototype software, etc. and maybe someday it will be useful, but it wouldn't have helped CloudFlare here if only because they needed no help noticing that all their systems around the world were crashing.

In our work on AI, at times we visited some high end sites, and in some cases we found some extreme, high up off the tops of the charts, concern and discipline for who, what, or why any humans could take any system management actions. E.g., they had learned the lesson that can't let someone just type in a new rule in a production system. Why? Because it was explained that one outage in a year, and the CIO would lose his bonus. Two outages and he would lose his job. Net, we're talking very high concern. No doubt CloudFlare will install lots of discipline around humans taking system management actions on their production systems.

Net, I can't blame CloudFlare. If my business gets big enough to need their services, they will be high on the list of companies I will call first!

<comment><USER59> _'in the words of the HAL 9000, "human error".'_

Except that it wasn't human error, at least not in the sense that the decision to enter the rule, or the rule itself was in error. The human error was with the bug in the Juniper firmware that caused this rule to crash this router, and arguably with the CloudFlare process that allows rules to be propagated to all routers concurrently, rather than segmenting the network and testing for success before further deployment.

<comment><USER60> Actually this outage report is a good example of compounding systematic errors. A among the things that went wrong: incorrect and impossible packet sizes were detected, the rule generator generated rules matching the impossible packet sizes, the human operator who looked at the rules and entered them in to the router didn't notice any problems, and finally the routers responded to the incorrect rules by starting to crash.

Had any of these steps not gone wrong there likely would not have been an outage. It was a combination of failures that caused it.

<comment><USER59> I don't disagree with you, but I was calling in to question the suggestion that the creation of the rule was the specific human error. By definition, every error you listed is a human error, as even if ultimately carried out but computers (routers), they were designed by humans.</comment></comment></comment></comment>


Ask HN: How do you come up with names for web apps? - <AUTHOR>

I've been staring at thesaurus pages for keywords I think apply to my app for too long without finding anything that jumps out at me - there has to be a better way. Has anyone found it? <comment><USER2> Why don't you tell us what your app does and we can help you come up with a name?

<comment><AUTHOR> Whoops - good point! With all my thesaurus fatigue I forgot to talk about my app!

It's basically another customer support application for web apps in the same vein as Tender (<http://tenderapp.com/>), but (hopefully) geared more towards supporting SaaS apps than anything else.

Edit: I should add that so far my best idea for names is Secret Sauce (because customer support is the secret sauce behind a lot of things), but it doesn't pass the "does 'Hey, did you log into Secret Sauce today' sound dumb" test.</comment></comment>


Ask HN: Y Combinator Wiki? - <AUTHOR>

I know that liking to things is cool, but it would be great if we could create more structured data like a wiki sometimes.
Best places to cowork
how to on taxes
start an ec2 box
etc.... <comment><USER2> Check out epi0Bauqu's site <http://www.gabrielweinberg.com/startupswiki/></comment>


Ask HN: So we think we innovate, But - <AUTHOR> Right from September 18 ', we were working on a product. Team spirit was as high as it could be. Lots of organizational resources were channelized on that product. Interviews, media publications and a lot of other stuff happened altogether, making us more interested. Two issues: There was a gap in the development of a few months due to some other important projects and we made a project which was not feasible exactly. Eventually, we fixed. A lot of innovation was happening until we came to know that Google and Microsoft are pitching the same project in the market. What should be our approach from here? We will appreciate all the tips/suggestions. <comment><USER2> I don't think you need to change anything. Not everyone wants to hitch their wagon to Google or MS. I'm one of them. I don't use any Google services aside from their search engine. I stopped using most all of Amazon's products and services too.

Why? Because there are always strings attached to that. For example, I don't want Google controlling or having access to my email.

Google's "App Store" is another good example. I read stories of developers getting booted off there almost weekly. Google provide almost no feedback as to why, and gives them no way to settle an issue.

I would highlight these kinds of issues in my marketing and promote personalized customer services.

<comment><AUTHOR> Thank you very much :)</comment></comment>


Ask HN: Would you hire me as a junior programmer? - <AUTHOR>

	Hello everybody, this is a rephrased version of an earlier posting I ended up editing I hope this doesn't constitute spamming. I'm finishing up my A.S. degree at a CC in the US. I have all A's in my computer related classes (Basic Java, Advanced Java, Basic C++, Database Administration, Systems Analysis and Design) and a few B's in other non-computer classes. I have also learned AS3 and Haxe. I am creating a social media site in Ruby that will demonstrate knowledge of Rails[0], and have created a few games (some used XML and JSON) using said Haxe and AS3.
The above examples show that I am a capable learner. However, I'm not quite sure how to best approach putting together a portfolio and contact potential employers. I'm in NY right now but will most likely be moving to SC for a year at least.
I understand that this is a bit of an open ended question, but I would love advice what to do at this point (I graduate in the spring and will be moving out to SC in late July).
[0]I'm currently using a private repo on bitbucket and developing on Arch Linux. <comment><USER2> What area of SC? If it is the Charleston area there are quite a few good opportunities for junior programmer. If you are planning on looking in or around Charleston be sure to take a look at the Charleston Digital Corridor[1] for some good leads. Also, I would be happy to speak with you as I have worked as a programmer in Charleston for the last 2 years and I love it. Feel free to get in touch: hello [at] jakejohnson [dot] me

[1][http://www.charlestondigitalcorridor.com/](http://www.charlestondigitalcorridor.com/)

<comment><AUTHOR> It's going to be very close to Charleston. Thanks for commenting. I'll be emailing you soon!

</comment></comment><comment><USER3> Unless you really have intentions of selling your site, you should just open all your code. There isn't much of a reason to keep it private, because potential employers can't view it.

But more to the point, you would have a better chance of getting a job in NYC than in SC. Just keep coding. Finish your projects too. Finish that site you are making. I would be more likely to hire you if you showed passion and interest in coding. And a big part of that for me is finishing the projects you start.

<comment><AUTHOR> Thanks for the great response! I'll most likely make the code for my site open once I'm finished with it, as I think it will make more sense once I have it all documented and tied together.

As far as finishing my projects goes, I have finished multiple small games. That said, the code is very sloppy since I worked on most of them before any formal training. Here are three games I've made for android using Haxe (for "Turnt" as a proof of concept) and Adobe Air (the other two games) [https://play.google.com/store/apps/developer?id=Zenkman](https://play.google.com/store/apps/developer?id=Zenkman)

Here is also another game I created using Haxe: [http://indiegames.com/2013/05/browser_pick_stockholm_adds_to...](http://indiegames.com/2013/05/browser_pick_stockholm_adds_to.html)

Once again, thanks for taking time out of your day to reply.

</comment></comment><comment><USER4> While your degree and grades count for something, it's more important that you build a solid understanding of programming concepts. You should also write code every day, for any reason that you can think of. Build up that skillset.

Relevant: [http://programmers.stackexchange.com/questions/26191/what- ar...](http://programmers.stackexchange.com/questions/26191/what-are-the- skills-you-look-for-when-hiring-a-fresher-newbie-for-developement)

<comment><AUTHOR> Thanks, that link is an interesting read.

</comment></comment><comment><USER5> The HN community is a bit unusual in that they're willing to consider programmers with no degree. But you should really just put in the extra 2 years, transfer to a state school and get a BS. It will pay off in the long run for your career. For most employers you would be at a big disadvantage having only an AS. Also keeps your options open for returning to grad school.

<comment><AUTHOR> Thanks for the comment. I'm under the impression that once you have your foot in the door so to speak, (a job that you perform well in) that a degree is for the most part a formality. Am I incorrect?

<comment><USER6> Speaking as part of the Bay Area/Silicon Valley software industry, -if- you can show a good Github portfolio and have a reasonable amount of prior experience, and both display obvious talent, you should be able to get in somewhere. Demonstrable experience > theoretical education.

Networking is important there, since recruiting tends to be much more about buzzwords and degrees than hiring managers are. Being able to get your resume straight to a hiring manager can be make or break.

That changes if you're talking software jobs incidental to other industries (financial, corporate IT, etc.) which is honestly most of them, statistically speaking. It definitely changes if you're talking consulting or field work, since clients want to be impressed by your credentials and your parent company will select on that. It also probably changes if you're talking outside an area that's already heavy in software industry.

Rule of thumb: jeans ok? You can probably manage without the degree if you're good. Khakis required? Not so much.

What the degree buys you, aside from access to the more conservative jobs, is some degree of assumption that you have some degree of training, as well as that you have enough follow-through to finish a degree. But even then, speaking for myself as a frequent hiring team participant, I'd be looking towards your internships, side projects, that Github portfolio, so forth.

Still, if you have the luxury of doing so, get the degree. It does make things easier.

<comment><AUTHOR> Thank you very much for the nuanced reply. My current plan is to go for my BS in CS at RPI or RIT next year if I don't find solid employment with a tech company before then.

Until either of those two things happen, I will be contributing to open source projects and creating my own projects while working other jobs in order to boost both my resume and skill level.

Great reply, thank you for your perspective.

</comment></comment></comment></comment><comment><USER7> I have some invites to StackOverflow Careers [1] that are just sitting. If you shoot me an email...you'll have to figure it out from my profile...I will send you one. It's a good place to post a CV.

[1]: [http://careers.stackoverflow.com](http://careers.stackoverflow.com)

<comment><AUTHOR> I sent you an email, thanks.

<comment><AUTHOR> I got it all set up, thanks!

<comment><USER7> Good luck.</comment></comment></comment></comment>


Ask HN: How would you assess the value of HN karma? - <AUTHOR>

Today a HN user challenged another to a wager to be paid in karma:
http://news.ycombinator.com/item?id=2004035
Which got me thinking - How would you assess the value of HN karma?
Yes $0.00, thanks. But how do you determine its value? <comment><USER2> I think it's kind of non-linear. You have a user with ~18000 karma wagering against a user with ~6000 karma. Moving from 18000 to 17000 karma would have no impact on rms' HN experience or general ranking. Similarly, paul is not going to get any big bump in going from 6000 to 7000.

Now, if paul was a new user with 5 karma, the 1000 point bump would open up options on HN that he did not previously have. It would also boost his perceived ranking slightly from an unvetted newbie to someone with roughly first-tier status.

So, the value of 1000 karma isn't some baseline value multiplied by 1000. It varies by user based on the relative impact it has to that user.

<comment><USER3> > Similarly, paul is not going to get any big bump in going from 6000 to 7000.

Actually, the bump from 6k to 7k would put you very close to /leaders.

</comment></comment><comment><USER4> To me it is the old Gentleman's bet of a dollar. More or less worthless outside of the fact that you must admit defeat, and surrender something. The thing surrendered doesn't matter or have any value beyond the symbolic.

</comment><comment><USER5> Why should anyone care? If you base your value of someone's opinion on their karma then you aren't really thinking for yourself. Someone with 1000's of karma points (even hundreds) just means they've been around a while, spent quite a bit of time, and had enough posts that resonated on the front page. Dig through posts that get few comments and you'll find some gems (they've just been overlooked).

And, really, what is the value of karma? Comment first and be insightful or funny - likely to get points if it hits front page. Comment late and be insightful or funny - likely to get a couple of points if it stays front page. Comment on anything not on front page - likely to be overlooked and even if you are insightful you may not get any karma, even from the OP.

Karma is kinda silly and even talking about it is sillier.

</comment><comment><USER6> The price of one karma point is upper-bounded by the cost of making it yourself. Since you could just create a sock puppet to upvote yourself (assuming newbies may vote -- haven't kept track of the catalog) in a minute, and the minute is worth $0 (because HN is a sweet waste of time), one karma point can't cost more than $0.

However, the price isn't linear. You can't boost yourself by a million that way because PG would notice. So million-point karma packages could actually have a nonzero monetary value.

<comment><USER7> Creating a sock puppet and upvoting yourself is against the secret HN rules. Please don't do it.

Sock puppet karma points are worth negative karma.

<comment><USER6> Self-upvoting is at least bad style. I'm not advocating to do it, just trying to estimate the minimum price if one wants to get karma somehow.

As for negative karma, does HN actively detect sock puppets to punish them?

<comment><USER7> Creating an account to upvote yourself is against the rules and HN has sock- puppet detection software that is not fully published in the open source release of news.arc. Reddit similarly has sock puppet detection software... all social news sites beyond a certain size need software like this or things like the home page being taken over by internet marketers happens.

</comment></comment></comment></comment><comment><USER7> Who wants to give me stuff in exchange for karma IOUs?</comment>


SXSW Cancels Gamer Panels After Threats - <AUTHOR> http://bits.blogs.nytimes.com/2015/10/26/sxsw-cancels-gamer-panels-after-threats/ <comment><USER2> The Gamergate/Anti-Gamergate movement _has_ to be a Sociology PhD Candidate's thesis topic. It's fascinating how utterly polarizing it is, I mean just look at the sort of comments we're seeing right here on HN, and we tend to be the levelheaded and civil ones!

Both "sides" are victims of their own success. In spite of whatever potentially valid points they have, the moment someone figures out which "side" you are on you are immediately lumped into one group or the other and considered completely guilty by association. And the folks that do that are so absolutely certain of how "vile" or "evil" you are, as certainly as if you'd come out as a Nazi or somesuch.

But if you could somehow make your argument without getting either label attached, someone might actually want to listen to you as though you were a rational human being.

<comment><USER3> > and we tend to be the levelheaded and civil ones!

I've seen a lot of things discussed on HN. I think that may be wishful thinking. Just because HN isn't full of straight out cussing and insults doesn't mean we're level headed or civil.

<comment><USER4> "levelheaded and civil ones" was clearly meant in relative terms. HN is not level headed or civil, we're just _more_ level headed or civil.

</comment></comment></comment><comment><USER5> #SavePoint on why their panel was cancelled: [http://www.theopengamingsociety.org/savepoint-dear- community...](http://www.theopengamingsociety.org/savepoint-dear-community/)

LevelUp on why their panel was cancelled: [https://twitter.com/randileeharper/status/658780409765994496](https://twitter.com/randileeharper/status/658780409765994496)

<comment><USER6> Which side is which here? I'm not really clued into the whole GamerGate thing...

I have seen Randi Lee Harper say some very unpleasant things, and some even more unpleasant things said about her. I know nothing about the Open Gaming Society, but reading their website it seems like a relatively sane organization.

<comment><USER7> SavePoint was to featured "several prominent GamerGate figureheads". The sponsoring organization generally claims to take "a neutral stance" on GamerGate. ([http://motherboard.vice.com/read/sxsw-cancels-gaming- panels-...](http://motherboard.vice.com/read/sxsw-cancels-gaming-panels-due- to-threats-of-violence))

Randi Lee Harper is noted for making tools to help victims of online harassment. Level Up panel description was "A panel from experts on online harassment in gaming and geek culture, how to combat it, how to design against it, and how to create online communities that are moving away from harassment."

The GamerGate crowd claim that they care about "ethics in game journalism" and that the abuse that feminist gamers experience doesn't have anything to do with them. So theoretically, there aren't opposing sides here.

</comment></comment></comment><comment><USER8> I get the liability arguments here, I really do, but people need to understand this is going to keep happening as long as people keep kowtowing to the demands of terrorists.

And I do not see that as too strong a word, either. Making threats against an event because it's against your ideology is the textbook definition of using fear to shape thought and policy.

These threats are never credible. It's high time to stop treating them as such.

</comment><comment><USER9> SXSW had the chance to take a stand for diversity here, choosing instead to show just how shallow they truly are.

edit: Added clarification of which "side" I'm on

<comment><USER10> What's hilarious about this whole thing is that I have no idea which "side" you are on by your comment. I've seen this exact sentiment expressed about SXSW's decision on both /r/KiA _and_ /r/GamerGhazi.

<comment><USER11> It's a mistake to think of it as "sides" rather than a small group of people with a persecution complex, and everyone else wishing they'd broaden their perspective a bit.

<comment><USER2> What's hilarious about this whole thing is that I have no idea which "side" you are on by your comment. I've seen this exact sentiment expressed about SXSW's decision on both /r/KiA _and_ /r/GamerGhazi.

<comment><USER11> Haha. But really, it's hard for that group to claim to be the silent, persecuted majority when nearly every major figure in the industry who has spoken about it and nearly every news outlet that has covered it has completely condemned it. The "movement" has even been placeholder villains on Law and Order SVU, and if that isn't the gold standard for general cultural sentiment, I don't know what is.

</comment></comment></comment></comment></comment><comment><USER12> So why not hold the panels in an undisclosed location and webcast them? Preferably for the public, not just conference attendees. Instead of giving the terrorists (for lack of a better word) exactly what they want turn it around and use their threats to garner extra publicity.

<comment><USER13> There isn't a better word, and we don't need one.

</comment></comment><comment><USER14> Why wouldn't they continue with the programming, and be ready to press charges against anyone who behaves violently?

<comment><USER15> Austin is located in the American state of Texas, which prides itself on gun ownership. Also, the SXSW conference has already had an instance of violence: a driver ran into a crowd of people and killed a few of them. While not intentional, I'm sure the programmers of this conference are eager to separate their brand from the idea of danger.

<comment><USER16> "Austin is located in the American state of Texas, which prides itself on gun ownership."

That doesn't really follow from the facts as a reason to be concerned about violence. Austin's crime rate is lower than that of most other major American cities, including ones such as Chicago which have heavy gun control. (As per Wikipedia, Chicago's murder rate for 2014 was 15.1, Austin's 3.5.)

<comment><USER15> I'm not sure the crime rate statistic is relevant in the context of the flood of threats being directed at the event. We're no longer talking about 'random crime' at this point. Also, if there's going to be a threat of violence, I'd prefer that threat not to happen in a state where people have easy access to guns; I understand that many people have trust in crowd-sourced defense (ala a well-armed populace) but I like to think the police and the event security are better choice for that.

</comment></comment></comment></comment><comment><USER17> What I find most interesting is how this thread is in position 89 with 38 points after only 2 hours.

There's some super heavy flagging going on to censor this whole thing.

</comment><comment><USER18> The whole Gamergate drama reminds me of the old saying: "The reason why campus politics are so dirty is that the stakes are so small." I mean seriously what kind of loser takes video games seriously? Or even worse, who actually cares about video game reviews?

<comment><USER19> If you're interested in interactive media -- and you've had an account on Hacker News for 8+ years, so let's assume so -- then you really should have at least a passing understanding of how video games work. Both technically and creatively. And as businesses. It's a fascinating topic -- and arguably we're going through a bit of a golden age!

If you're just interested in making fun of nerds for liking video games... Again, you've had a Hacker News account for 8+ years. Might want to watch you you call "loser" for liking stuff related to technology.

(8+ years? Unless you first created your account when you were 10 years old, you're literally an adult who wrote the comment above. Wow!)

<comment><USER18> Ha ha. I used to be a hard core gamer. Then I grew up, moved out of my parents' house, and got a life. There's nothing wrong with playing games as a hobby. The losers are the ones who actually care about games and take them seriously.

<comment><USER19> Whelp, you're missing out on some interesting stuff, man. Happy to hear you finally moved out of your parents' place!

</comment></comment></comment></comment><comment><USER20> Wow, didn't see that coming. /sarcasm

</comment><comment><USER21> What's frustrating about this whole situation is how deceptive the "extremists" have been on both sides (GamerGate and Anti-GamerGate) by making false-flag attacks. You just can't tell who is faking what now, and it's making reasonable discussion between the two camps impossible, further polarizing everyone.

</comment><comment><USER22> Threatening violence to noplatform your ideological opponents is a pretty standard tactic out of the SJW playbook.

See: [http://www.polygon.com/2015/5/3/8539733/gamergate-bomb- threa...](http://www.polygon.com/2015/5/3/8539733/gamergate-bomb-threat- washington-ggindc-milo-yiannopoulos-christina-hoff-summers)

[http://www.polygon.com/2015/8/16/9161311/bomb-threat- shuts-d...](http://www.polygon.com/2015/8/16/9161311/bomb-threat-shuts-down- spj-panel-discussing-gamergate)</comment>


Are hackers heroes? - <AUTHOR> http://www.nybooks.com/articles/archives/2012/sep/27/are-hackers-heroes/

<comment><USER2> No, hackers in general are not heroes though they (or I should say we) seem to have huge egos and would like to think they are heroes. There's nothing heroic about jail breaking phones and tablets. There's nothing about being a hacker that is in itself heroic. There are however a number of hackers whose work is based on idealistic beliefs and/or for what they believe is for the good of the public and what they do could be considered heroic and so they may be labelled heroes but are hackers heroes? No. I don't like the the fact that someone would try to say they're heroes and the fact that this article landed on the front page here smacks of hacker egotism. Are doctors, school teachers, stock brokers, entrepreneurs, or barbers heroes? Of course not! You could certainly twist your reasoning like a pretzel to argue they are but the truth is these are just people and not all people are heroic and your profession doesn't make you a hero. There are doctors who are in it for the money and there are doctors who want to make a difference. Neither type is heroic in and of itself. To simply go against the grain, question the status quo, and work against the current state of things is not necessarily heroic.

Hackers are not heroes. Certain individuals out there are heroes. They aren't all hackers. They're people from all walks of life and professions, at varying skill levels who do something at great risk, who often sacrifice something, and do something for the good of one or many people. The term hero can be very subjective and I am not comfortable labeling a group of people or even a subset of a group of people as heroes. The fact that someone wrote an article like this just leaves a bad taste in my mouth. Hackers, you're not heroes. Only heroes are heroes.

<comment><AUTHOR> I think you're missing the point of the article at least a little here, although I agree with almost all of your points. Granted, hackers like any other group aren't all heroes or not heroes, and as you say the term is a loaded one already. But if you can define heroes as "people from all walks of life and professions, at varying skill levels who do something at great risk...for the good of one or many people", then there are certain examples in the article that fit that description. Heck, even the kids stealing credit card details are benefiting themselves, which actually conforms to your definition.

In certain cases I think black-hats are completely justified. I don't see what's wrong with DOS attacks on an organization like the FBI, a well known perpetrator of torture internationally. Ok, so one could argue that most DOS attacks are merely mischievous and aren't politically or morally motivated, but some are, and it's usually a case of the little guy (small groups of citizens) against the big guy (governments), and one could see this kind of hacking as a force tipping the balance in the individual's favour.

</comment></comment><comment><USER3> Hacker are not heroes as a rule, no sterotype is. A hacker and were talking non-tabloid version of the term I presume are people who question the anwears as much as the initial question and have the tenacity to seek out those answears to the questions nobody else bothers to ask.

A hero is somebody who goes further than others even imagined, it is always a label bestowed upon somebody by others. Hacker is a mindset and when it is bestowed by others is generaly refering to some electronic vandalism.

Using a bic pen to perform a trachyotomy so somebody can breath, that would be a hero hacker if such a definition was needed.

</comment><comment><USER4> This discussion is framed very weirdly, almost to the extent of meaninglessness because the language is so imprecise. Asking "is hacking good" is like asking "is evolution good for people when it produces everything from terrible worms that blind people by living and eating their eyes to fantastic bacteria that allows us to digest all kinds of crazy foods?"

Systems always have weaknesses and systems always change. Sometimes we improve the weaknesses for fun or profit or usefulness, sometimes we attack them for fun or profit or malice.

<comment><USER5> Your biology examples raise the idea that it would make sense for the legal system to distinguish between types of computer intruders. As vaccinations used to work on the principle of exposing people to milder strains of a virus, it'd be likely to be beneficial to make restricted forms of hacking legal (largely based on non-sharing of any non-public information from servers). The devil would be in the details of course, but it might be worth considering.

<comment><USER4> I would agree with this suggestion. The article categorizes everything from modifying phones to illegal and wrong activities such as theft and spying as hacking. Things will always be used for purposes different from their original intent because humans are resourceful and care about their own wills. Neither humans nor nature care what the original creator's intentions might have been- our eyes may be "intended" for sight but worms will evolve to use these eyes as food and housing- and it's nonsense to label these worms or their evolution as good or evil. Hacking is an idea that will always occur because it's part of nature, which has no sense of justice. Morality only comes in with how we humans use our ideas. For example, the making of a knife from a toothbrush is not illegal, even if it offends the sensibilities of the toothbrush designer, but the use of this knife to harm another person is illegal.

</comment></comment></comment><comment><USER6> Confront and Conceal is an odd choice given the other three titles. There is only one or two chapters in C&C that are relevant and they add very little to the debate other than providing some "ripped from the headlines" context. The reviewer should have gone with "This Machine Kills Secrets."

</comment><comment><USER7> Yes, we are. "Crackers", on the other hand...

<comment><USER7> although I guess we can't say "crackers" anymore... damn young people.</comment></comment>


A slideshow with a virtual tour of earth -- what content would you like to see? - <AUTHOR> http://www.slidetrip.com/

<comment><AUTHOR> I have created this cool on-line app that I love. Just wondering if you all think it's cool too. (slidetrip.com)</comment>


Google Ordered to Turn Over YouTube Viewer Data to Vaicom - <AUTHOR> http://consumerist.com/tag/privacy/?i=5021983&t=judge-orders-google-to-turn-over-all-youtube-user-data-to-viacom

<comment><USER2> I am surprised this story didn't get any up votes.

They are invading privacy and nobody comments about it here in Hacker News?

<comment><USER3> Maybe you should define the "they" you're speaking of?

I've noticed a disturbing trend on the internet with this case specifically. Google did not OPT to give this data to Viacom. They were ordered to by the U.S. Government. Instead of accusing Google of somehow now being evil, maybe we should be complaining about a court system that doesn't understand technology at all.

Until the courts have a grasp of technology, we will continue to see these overreaching orders on a regular basis. The courts just don't know how to be more specific, so they grant access to everything.

<comment><USER2> "They" I didn't mean Google</comment></comment></comment>


World Trade Center Towers Fill Slowly in Shift to Tech - <AUTHOR> http://www.bloomberg.com/news/2015-01-07/world-trade-center-towers-fill-slowly-in-shift-to-tech.html <comment><USER2> For 4 years I've lived about 5 blocks from the WTC compound and walk by it daily on my commute and as we move around the neighborhood.

When we first moved here we were surprised at all the openings, it seems most of the financial companies have folded up or left the area and it's being rapidly developed as a residential area.

The building I live in is something like 9 years old and all around us new buildings are going up with signs all over them about new residential housing. Still many other buildings are being refurbished to support residential housing and "fully furnished" housing.

When I heard Conde and Media Math were moving in I told my wife "Here it comes Silicon Valley on a single campus". I recently learned of another very large tenant in Media and Tech that's moving in (can't share the name) along with several others related to the media and tech scene in NYC.

My wife and I had been considering moving to another part of the city until we realized I'd have a 10 minute walk to most of the tech and media companies I'd either work for or consult for.

Maybe the buildings are filling up slowly, but if they keep filling with tech and media companies the surrounding residential spaces will fill up quickly as the employees snag walking distance housing. This could create a really nice synergistic geographic location for tech and media innovation.

I know this won't be a popular comment since most of HN is valley based, I spent 5 months of 2014 in the SF area, working for a start-up and consulting for clients in the valley. I burned so much time on the road my wife offered to commute with me so I could be in the carpool lane. Our company sold and I came back home to NYC. I've enjoyed being back home.

One of my frustrations with "The Valley" is having to own a car to really be able to move around. Yes there is some transit but nothing like NYC. I lived most of my life in Los Angeles so I am overly aware of how horrific the car culture becomes. You lose so much time in the car, you're less productive, you have less family time and you're stressed and burnt before you even sit in your chair in the morning at work. When you have to start your own bus company to get your employees in and out of work something is amiss.

I am hopeful that downtown could become the next challenger to the valley and do it with less people stuck on a freeway somewhere.

<comment><USER3> Every 5 minutes further from the office is about 40 hours of commuting over a year. When I look at a job that's further away from my house it helps me to weigh the cost/benefit of losing a week or more of vacation.

<comment><USER4> While that puts commute in perspective, time is not fungible like that in reality. 5 minutes on the margin won't make much of a different. Will you really take an extra week of vacation if move 5 minutes closer? No. Let alone the huge time cost of changing workplaces or residences.

</comment></comment></comment><comment><USER5> I think there is a major bias against the towers from the finance/insurance industries in NYC. Too many of their colleagues and friends were killed or impacted the last time around, like working in a graveyard. Here is a harrowing story from someone who was working there that day:

[http://www.wernerf.com/article/kamran.html](http://www.wernerf.com/article/kamran.html)

It is plausible that tech firms would fill in the gap as they don't have the same associations, the buildings are brand new and they can build out or subdivide as they see fit. Without huge demand for the space, I'm sure the tech firms are getting awesome deals, well south of the 50-60 per sqft quoted here when all incentives are factored in.

I'd even guess this is an example of Paul Graham's "The Submarine" in action - PR people trying to hype up demand for the building.

<comment><USER6> In addition to that, the firms have all moved away and figured out a solution for their office space in the past _14 years_. It's not like the financial firms were waiting in hotels all that time for them to build a new tower. The demand will have to come from companies looking for new office spaces, so generally growing or shrinking companies.

<comment><USER7> This…many of the major firms either moved to midtown, or to Jersey City.

</comment></comment></comment><comment><USER8> While the article tries to make the point that the tower is filling slowly, it fails to do any useful analysis to demonstrate this. Is it filling slower than the original towers? Is it filling slower than comparable office space, in NY or elsewhere? The writer never bothers to find out or inform us.

<comment><USER6> That's a very good point. The Hudson Yard office towers are similar in size and amenties, with a bit worse transit access than the WTC. They are also having a hard time filling the space.

In general NYC construction is mostly residential (especially luxury), it seems like there is more demand for that at the moment. Most of the tech companies and startups are housed in the space above downtown but below proper midtown, in interesting older buildings. I generally think of the "startup" zone as Madison Square Park down to Soho.

</comment></comment><comment><USER9> > The building -- still known to many by its former name, the Freedom Tower -- > has faced criticism in recent months, including a negative architecture > review in the New York Times by Michael Kimmelman, who wrote that the > skyscraper "looks as if it could be anywhere, which New York isn't." > Comedian Chris Rock, in a Saturday Night Live opening monologue, called it > the "never-going-in-there-tower, 'cause I'm never going in there."

Both of those criticisms are so weak that I wonder how they made it into a Bloomberg article.

<comment> k-mcgrady The second one isn't even a criticism really - it's a joke. Sure it could be a real critique (still not a great one) but it's more likely he thought, "this sounds funny" and said it because _he 's a comedian_.

</comment></comment><comment><USER10> > At that pace, the towers wouldn't reach 95 percent occupancy until 2019, > almost two decades after the 9/11 attacks leveled the first twin towers.

How is it that Manhattan residential and Manhattan commercial are so out of whack? As an example, Lincoln Towers, which from the outside looks like a public housing project. That being said, it sells for $1,400 square foot.

[http://en.wikipedia.org/wiki/Lincoln_Towers](http://en.wikipedia.org/wiki/Lincoln_Towers)

And we have these nice new commercial buildings that will take years to fill.

<comment><USER11> There are a lot more residential buyers and renters than commercial renters.

Take the crazy supertalls going up on 57th St right now - those homes are actually surprisingly hard to sell, and those buildings also take years to fill purely by virtue of the fact that the number of qualified/interested buyers in the world is so low you can fit them all in the same conference room.

Commercial space is in high demand in Manhattan, but companies also have relatively narrow parameters in the spaces they're able to rent (company size, location, nearby facilities, etc), which limits the number of potential renters.

<comment><USER10> > There are a lot more residential buyers and renters than commercial renters.

That's my point. If this is the case, why do they keep building office buildings when the city actually needs apartment buildings?

<comment><USER11> Because the space is needed. The speed at which a market moves is not a measure of its size nor its supply/demand balance.

Commercial real estate vacancy rate in Manhattan right now is quite low compared to historical trends - don't let the stories of megatall commercial projects fool you. Cost per square foot has been rising steadily throughout Manhattan and is now actually quite high.

New buildings always take a lot of time to fully lease out due to structural inefficiencies and the more constricted customer base - the Empire State Building wasn't profitable for decades after it opened. This doesn't mean that the office space isn't needed.

There are a lot fewer commercial renters, but they occupy a lot more space for a lot more time. Your typical New Yorker isn't, for example, renting a 30,000 square foot space on a 10-year lease.

</comment></comment></comment></comment><comment><USER12> This seems to insinuate that financial firms are not as numerous or maybe not as big as before. Does anyone know whether the financial sector has significantly grown or diminished from 2001 to now?

<comment><USER7> Finance is still a dominant sector in NYC but has been dropping since the late 1990s (accelerated post 9/11, but there's been a long term trend of back- office jobs being eliminated or moved out of the metro area).

</comment></comment><comment><USER13> I've been working in the building for about a month. Working at Conde, I'd love to see more tech / publishing companies move in.</comment>


Facebook under fire as political ads vanish from archive - <AUTHOR> https://www.ft.com/content/e6fb805e-1b78-11ea-97df-cc63de1d73f4 <comment><AUTHOR> Article is paywalled, so here are the key bits:

 _Tens of thousands of political ads have gone missing from Facebook's public archive less than 48 hours before the British general election, raising fresh questions over measures introduced to address transparency concerns in online political advertising...

...On December 5, Facebook's ad library, which includes adverts shown on its photo messaging subsidiary Instagram, showed a total of just under 180,000 classified as political in Britain since October 2018. But on Tuesday, that number had suddenly dropped by a third, to fewer than 120,000 ads.

Facebook declined to say what had caused the issue, or whether the ad data were irretrievably lost or could be recovered. But a company spokesperson said: "We're aware that people are having trouble accessing the ads in the ads library, and we're working to fix the issue as soon as possible."...

...The data lost by Facebook did not appear to be limited to any particular party or subset — ads were missing from all the main parties' pages._</comment>


ITC says HTC violating two of Apple's patents - <AUTHOR> http://news.cnet.com/8301-1035_3-20079905-94/itc-says-htc-violating-two-of-apples-patents/

<comment><USER2> Slightly more info by following this link:

[http://allthingsd.com/20110715/itc-rules-htc-violated-two- ap...](http://allthingsd.com/20110715/itc-rules-htc-violated-two-apple- patents/?refcat=news)

Patents in question:

[http://www.google.com/patents/about/5946647_System_and_metho...](http://www.google.com/patents/about/5946647_System_and_method_for_performing.html?id=aFEWAAAAEBAJ)

[http://www.google.com/patents?id=nCYJAAAAEBAJ&printsec=f...](http://www.google.com/patents?id=nCYJAAAAEBAJ&printsec=frontcover&dq=6,343,263&hl=en&ei=3rUgTojDCtHXiAKxsaTOAw&sa=X&oi=book_result&ct=result&resnum=1&ved=0CCgQ6AEwAA)

Are both of these really as obscenely broad and all-encompassing as they look? WTF, USA?

</comment><comment><USER3> At least one of these patents is apparently fundamental to Android which means Google needs to get off their ass and step up.

I believe the timeline is 60 day review then judge rules (which could go the other way, usually doesn't change much) and of course there are appeals.</comment>


Do I trust these founders with this market? - <AUTHOR> http://soundboy.tumblr.com/post/20401856991/do-i-trust-these-founders-with-this-market

<comment><USER2> Beautiful and deserves to be widely read.

It goes well with something Fred Wilson said: <http://news.ycombinator.com/item?id=2721767></comment>


NoSQL is a Stupid Name - <AUTHOR> http://java.dzone.com/articles/nosql-stupid-name <comment><AUTHOR> Amusing set of definitions from the article:

Cloud - Fire your systems people and ditch your comms room!

Big Data - Parse Twitter in order to learn how to read your customer's minds!

NoSQL - Stop paying Oracle!

Functional - We couldn't get good enough at mainstream programming languages so we switched to something more difficult!</comment>


Input methods for Unicode characters in editors and the shell - <AUTHOR> http://doc.perl6.org/language/unicode_entry <comment><USER2> I love OSX's mnemonics for entering non-ascii. I don't know if they differ from keyboard layout to keyboard layout, I use US English.

Option-e then a vowel produces that vowel with an acute accent. I think "español". (Oh, `option-n, n` produces ñ). Option-! produces ¡ (upside down bang used in spanish).

Option-u then vowel produces vowel with umlaut/diëresis ö ï ä ë.

Option dash is an en-dash and shift-option-dash is an em-dash. Option-< is ≤ and option-> is ≥. option-= is ≠. shift-option-2 (ie option-@) produces the euro sign € (I wonder why it isn't option-$, it was probably already used for something else, oh well, only so much room for mnemonics).

Those are the only ones I currently remember, because those are the ones I use anything close to regularly, but many other non-ascii characters have similar mnemonics using the option key, if I need one I look it up and use it.

Works in just about any software at all, including the terminal.

I don't know why other OS's haven't done similar. Maybe lack of that useful extra 'option' key, the other meta keys may already be taken for other purposes in other OS's.

<comment><USER3> The compose key is far more intuitive for infrequent use [1]

To type ñ, type Compose, ~, n.

To type →, type Compose, -, >.

To type €, type Compose, C, =.

To type —, type Compose, -, -, -.

To type {vulcan salute, which HN strips}, type Compose, L, L, A, P. (Actually, this doesn't work for me, although it is present in /usr/share/X11/locale/en_US.UTF-8/Compose.

[1] [https://en.wikipedia.org/wiki/Compose_key](https://en.wikipedia.org/wiki/Compose_key)

<comment><USER4> Seconding this. No other method of inputting special characters feels nearly as intuitive as the Compose key.

I personally like to map Compose to the Menu key, because the Menu key's default behavior is totally useless.

<comment><USER5> The menu key's functionality is also usually available through Shift-F10 as well, so swapping a more useful key into its location doesn't lose you anything.

</comment></comment></comment></comment><comment><USER6> I've seen some complaints about this, but the existence of Texas versions of the operators makes this absolutely fine in my eyes.

I don't see why people would begrudge the ability to use the 'proper' mathematical symbols just because they're slightly harder to type.

Of course, it's another situation where Perl 6 is just crazy, but crazy awesome :)

<comment><USER7> I instinctively feel like I'm going to run into some stupid problems, but I blame that on the extreme difficulty in 2015 of getting UTF-8 to work consistently with the mess of locale variables, terminal emulator configuration, vty multiplexers, editors, source control systems, MIME types, and who knows what. Pretty much every time I've put Unicode symbols in source code I've ended up regretting it later for some frustrating reason. There are always solutions though... and kudos to Perl 6 for giving it a shot.

<comment><USER6> I'm sure you're right, the REPL uses Linenoise which is single byte character only for example.

The only way to fix these problems though is to increase their use, so ultimately the end goal will be worth it. That seems to have been the Perl 6 mantra for a decade or so.

<comment><USER8> Tip: don't install Linenoise and use rlwrap instead. You get unicode, history across sessions, and if you tweak it right tab completion.

</comment></comment></comment></comment><comment><USER9> Inputting Unicode in X Windows is easy. Just type:

      setxkbmap -option compose:ralt

Now, right Alt (AltGr) is your compose key.

Then, for example:

      à is Compose + ` + a
      ö is Compose + " + o
      ß is Compose + s + s
      ç is Compose + , + c
      € is Compose + = + e

In Emacs, the easiest way to input Unicode directly is to set read-quoted- char-radix to 16 in your .emacs. Then use C-q (typically bound to quoted- insert) to input a glyph directly via hex code point.

      ∀ is C-q + 2200
      λ is C-q + 3bb
      ∃ is C-q + 2203
      若 is C-q + 82e5
      世 is C-q + 4e16

An easier way is to dial up a character map program like gucharmap and copy and paste.

<comment><USER10> I've always wondered why it wasn't the default in Linux to have a compose key configured, and I used to have it set in the distant past.

I just found it's editable in the Gnome settings editor. Browse to Keyboard->Shortcuts->Typing. Then click the "Disabled" next to Compose, which will allow selection of the compose key. The directions at the bottom of the settings window are simply incorrect, or don't apply to this particular screen, I dunno. Regardless, it works well for most common characters (the only language I regularly type with need for them is Spanish, so it covers all the bases there).

</comment></comment><comment><USER11> Or you could use linux and XCompose and set up easily recallable shortcuts for frequently-used symbols that don't have them already. I think trying to use and remember different input methods in different apps is insanity.

<comment><USER12> That's what I came here to say. I can't believe the use of the compose key isn't more common. I just remap caps lock (which I don't like) to compose and kill two birds with one stone...

<comment><USER6> On a lot of Linuxes you can use alt-gr for third level keys too, which is handy for «» for example (altgr + z, x) or "" (v, b)

</comment></comment></comment><comment><USER13> The "bash" thing seems to actually be a GTK+ thing (and that's how Wikipedia documents it), and probably only works in gnome-terminal and a couple of other X11 terminals. I can't seem to get this to work in iTerm2 + OS X bash nor iTerm2 + ssh + Debian bash.

On the other hand, I guess I'm running OS X so I merely need to re-memorize all the Mac mnemonics that I knew back when I used OS 9, and have now forgotten in favor of GNU screen's digraphs....

</comment><comment><USER14> On OS X, Ctrl-Cmd-Space brings up the Emoji palette. From there you can also search for other kinds of Unicode characters.

<comment><USER15> The built-in panel has gotten better in recent versions of OS X but still has some quirks. It doesn't copy the character into the clipboard (it copies the full description along with it, and even that requires a right-click), but you can double click things to paste a character into the app that previously had focus, most of the time that works pretty well.

Before I found out about the built-in panel I wrote a menu bar app[0] that does copy directly to the clipboard, and I still use it since that's what I need more often than the "paste into previous app" behavior of the built-in panel.

[0] [https://itunes.apple.com/us/app/codepoints/id499161264](https://itunes.apple.com/us/app/codepoints/id499161264)

<comment><USER14> Copy to the clipboard? Ctrl-Cmd-Space brings up a panel that directly inserts text.

<comment><USER16> Provided the application you're using supports it.

For example I just needed to insert some UTF-8 characters into MacVim, which doesn't support this palette. So I had to:

\- switch to Safari

\- open the palette and find the character

\- make sure a text field is focused *

\- double-click to insert *

\- finally copy the character *

Those last few steps wouldn't be needed if one could simply copy just the character from the palette.

(Disclaimer: normally I would just search google for "Unicode _____", but wanted to try this after seeing it mentioned)

</comment></comment></comment></comment><comment><USER17> I'm using tcsh, and recently found out that it is practically impossible to enter curly-braces { } into it. For example:

        echo "{"
        echo '{'
        echo "\{"
        echo '\{'

all fail. This makes tcsh totally unsuitable to work with, for example, JSON expressions.

<comment><USER18> I just opened tcsh and all of "{", '{', and \\{ (an option you missed) work.

<comment><USER17> Hmm, okay, upon closer inspection, you are right. The problem I have is that I have the following definition in my .cshrc file:

        alias precmd 'echo `date` "#@#" `pwd` "#@#" `history 1` >> ~/.complete_history'

This basically saves all my commands to a file (which is very handy, by the way).

So somehow this interferes with the curly braces. I put the history part in double-quotes, and now it works:

        alias precmd 'echo `date` "#@#" `pwd` "#@#" "`history 1`" >> ~/.complete_history'

Thanks for helping me solve this :)

</comment></comment></comment><comment><USER19> XCompose is great for this (oddly, it's completely missing from the page): Compose+th = þ, Compose+"u = ü &c.</comment>


Ask HN: What's the difference between building features and a user experience? - <AUTHOR>

It's often said that companies should focus on the user experience, not the features.  What exactly does that mean?
Features are built so that users can do more things which will presumably improve their experience.
It is just a matter of the coherence of the vision at a certain level of abstraction?  (Only care about feature insofar as it furthers a certain more abstract issue of the customer experience?)
Maybe I answered my own question.  I invite people to pontificate.  Thanks. <comment><USER2> They go hand in hand frankly. Also I feel they both deserve equal and adequate attention. If you build something and it is missing features you are failing, same as if you build something and it lacks a user experience you are failing. However which is more noticeable? I would say it depends on the feature.

Ultimately, whether smart or dumb an end user will know if the user experience is lacking. If a feature is missing though a user might or might not notice. Depends on the feature.

Features are built to provide one part of the user experience, the UI itself is what provides the overall user experience which is to an extent dictating by the features required.

Make sense?

</comment><comment><USER3> One way of distinguishing between the two would be that user experience is the cognitive interaction with features. If you start with a basic web app, then add a user chat feature and then a group chat feature, ux would be how new and existing users interact with those features and your entire application.

Put another way, user experience encompasses the design aspects and the behavioral aspects of an app, as where features include the different components that make up the overall user experience. Its the interaction with the existing features that compose the ux.

</comment><comment><USER4> People complain about two things:

First, they complain about things that ought to work well but don't -

"I can never find that option"

"it crashes when I try to do this"

"it's counter-intuitive to need this"

"the defaults are not sensible"

"it takes too many button presses to do a common task"

Then they complain about things that they want to be able to do, but can't -

"it should support multiple timezones"

"I want to be able to sync across my devices"

"why can't I export it to another format"

"I want more privacy control"

The point is that no matter how much you work on the second list, if there is notable stuff on the first list _your users will be unhappy regardless_

I am not convinced you can completely evaluate by yourself the importance of list A against list B. I really recommend polling users in controlled and well-thought out ways.

</comment><comment><USER5> Being that this discussion board is most frequented by those involved in startups, I would posit the meaning behind "UX, not features" to actually be: Focus on maximizing the experience of a small number of features over doing an adequate job on a large number of features.

As Paul Buchheit stated, it's better to make a small number of users love you than a large number of users kind of like you. To do this, maximize the preferred, smaller feature set of that excited minority initially, and then expand to encompass the majority only after you've captured that minority.

</comment><comment><USER6> > It's often said that companies should focus on the user experience, not the > features. What exactly does that mean?

It means focus on improving the user experience in the already existing features, instead of just adding more and more features that don't fit together.

Good user experience is somewhat invisible, and it seems rather simple when you look at the results. Think of multi-touch on the iPhone.

Good user experience often requires cutting features out, which makes it hard to get right if you're into the "let's add more features" mentality.

</comment><comment><USER7> Building a good user experience (in my opinion) involves: 1\. A research period to determine which features your users need and 2\. A brainstorming period to determine how to implement those features in a way that makes them easy to learn over the short term, and easy/quick to use over the long term.

So to answer your question, user experience is more about the process of how to build good features.

</comment><comment><USER8> Perhaps this isn't quite the right question. patio11 says <http://news.ycombinator.com/item?id=1774112>

</comment><comment><USER9> Design for the novice, configure for the pro.

</comment><comment><USER10> who says don't focus on features?</comment>


Yoda still standing: Pentagon futurist Andrew Marshall, 92, survives budget ax - <AUTHOR> http://www.washingtonpost.com/world/national-security/yoda-still-standing-office-of-pentagon-futurist-andrew-marshall-92-survives-budget-ax/2013/12/04/df99b3c6-5d24-11e3-be07-006c776266ed_story.html <comment><USER2> I bet he didn't see that coming. _ahem_

Futurology is a really interesting field. I was fortunate enough to attend a workshop by the reknowned trendspotter/futurologist Magnus Lindkvist a while ago. It was brilliant. If you have half an hour to kill, this talk is the essence of it: [http://vimeo.com/44026423](http://vimeo.com/44026423)</comment>