Permission to Suck - <AUTHOR> http://www.kadavy.net/blog/posts/permission-to-suck/ <comment><USER3> This is where my startup failed -- We made the decision to work in a new technology (then another!) but I was afraid of writing code I was less than proud of.

I then proceeded to learn what I could of best practices, spending far too long experimenting with the language and far too little actually doing. In the end, we ended up with two half-baked infrastructures in two different languages and reached the trough of disillusionment before anything we could show to be disillusioned about!

It's something that I've learned in other parts of my life, especially through writing, but have never quite managed in coding. The primary tenet of National Novel Writing Month is that to finish a rough draft of a long fiction work, you must, starter to professional, give yourself permission to suck, to write something you know is sometimes horrible. You do this so that you have a canvas to fix during editing, to find nuggets that you love, and can revise, and can work with.

Intellectually, I know this about programming as well, but I always have this twinge of guilt when implementing it, and go back to my old ways. The irony is that in larger projects, this can be a useful quality. Having someone on the team that can go back and look at the code, showing exactly where we need to improve before we ship is useful. In solo projects it is often disaster.

At least for me. :)

<comment><USER4> I suffer from this as well. It's called Analysis Paralysis. Now, I just need to figure out the best way to stop it.

<http://en.wikipedia.org/wiki/Analysis_paralysis>

<comment><USER5> Best fix I've found is iterative development. No better way to learn something then to try to directly apply what you're trying to learn. Build it. If it sucks, build it again. Repeat as necessary till it's 'good enough'.

</comment></comment></comment><comment><USER6> Do you know what I like about HNs compared to other sites (e.g. Reddit, /., etc)?

It is constructive. There are a bunch of people who are interested in self improvement, trying things, failing, and then picking themselves up and trying again.

That's rare. You don't find that in many communities. Most of the time it is just negativity and "you can't," "you shouldn't," "here are all the flaws," etc.

Hopefully this grouping of people and ideas will help all of us in some way, they do say that it is the people you surround yourself with...

<comment><USER7> Well reddit is also a lot of this when you dig in the smaller communities. Sure, if you stay on r/AdviceAnimals you are bound to be disappointed ;)

<comment><USER6> There are tons of cool communities but none that I've found yet about starting a small business or similar. If you know of any along the same themes as HN then I'd love to join them.

<comment><USER8> /r/startups /r/Entrepreneur /r/smallbusiness

... of course, you'll find plenty of reposts from HN.

</comment></comment></comment></comment><comment><USER9> This is a well-worn piece of advice, but it's well-worn for a reason - <USER2> it's incredibly valuable.

Observing a lot of my friends over a lot of years, I'm starting to suspect that the willingness to fail, and particularly to fail in public, is possibly the key differentiator for people who achieve impressive things.

(I'm currently learning a new skill myself. I was sucking very badly, I'm now just sucking pretty badly, and I'm enjoying the progress.)

<comment><USER10> > particularly to fail in public

This reminds me of when I learned to dance Salsa, and more recently, Tango.

It's a far more public failure than writing a blog post. Your body – the very thing that experiences the embarrassment of failure – is your instrument when dancing. To add insult to injury, you're failing in front of a never-ending stream of women.

But, thankfully, sucking is temporary and life is long.

EDIT: I'd like to add that life is much, much, better when you know how to dance.

<comment><USER9> Yes. I had a similar experience when I tried to learn to dance, some time ago. The "right, now go and dance freestyle with Random Attractive Woman #1" moment was particularly terrifying.

(I still can't dance very well.)

</comment></comment></comment><comment><USER11> Many branches of meditation are based on the idea that there is no way to suck at meditating. Simply observe the thoughts arise and fade away.

If you are trying to succeed at meditating you are working against your self.

<comment><USER12> This is why I hilariously love meditation. The goal is to suck at meditating, because there is no goal.

</comment></comment><comment><USER13> In a nutshell: humans learn, live, and eventually succeed via a simple try- fail-try-again process.

If you can't suck, you can't fail. If you can't fail, you can't learn, you can't progress.

In most corporate places, you're not allowed to fail. Most will refuse changes, even if it's for the better, because, it may suck. It may fail.

So yeah. If you wanna be great, you gotta suck.

</comment><comment><USER14> I found this out on my own in some sense, but I still have trouble putting it into practice.

I was taking guitar lessons and I would always be afraid to bring in side tunes I had been working on because I thought they were terrible (and they are) but it's tough to give yourself permission to suck when doing nothing hurts a lot less.

<comment><USER10> I've been playing guitar for 13 years, and I still am not very good.

A breakthrough for me though, was when I stopped obsessing over getting every note right, and instead focused on strumming the right chords. Then, I found playing much more enjoyable, and it was easier to play whole songs and sing.

Once I got that down, I was able to move on to more complex things. Like many things, it helps to embrace your suckage with musical instruments.

<comment><USER15> Randomly connected, but the game Rocksmith takes this approach. It'll give you points if you get the strumming correct, but are fretting 2/5 notes correctly. There's a huge focus on "get as much as you can right now, but we're going to keep throwing this at you until you get it right."

</comment></comment></comment><comment><USER16> No! Meditation is not about thinking about nothing! It's about focusing on your breath. Think only about breathing.

That sounds boring but manageable. 'Thinking about nothing' seems impossible and stupid at the same time; people who do the disservice of saying that should be slapped.

The point of meditation is to learn how to control your thoughts, specifically in being able to realize that your are choosing your current train of thought, and that you can switch it at any time. If you are stuck worrying, or being sad, or unable to see the richness in something, it's most likely because you can't properly focus on the only thing you'll ever have in life: this moment.

<comment><USER17> There are many goals to meditation. Sometimes it starts with focusing on your breath. Sometimes it's in the form of chanting. Depends on the school of thought, the individual, the goal (if you have one). It's not valid to say "No!" as though what he's proposing is wrong.

<comment><USER18> No!

While you & the author are technically right, batgaijin is saying that it's bad advice for beginners to be told "think of nothing".

</comment></comment></comment><comment><USER19> This especially resonated with me. I wrote about a similar topic [1] a few weeks ago, but being a college student - I love this "permission to suck" idea. It gives me the ability to stop being ashamed of my crappy projects, knowing that I'll get better. I find most college students simply don't work on stuff outside of school - probably because they don't give themselves permission to suck - and it really affects their abilities in the long run.

[1]: <http://jscal.es/2012/10/12/you-oughta-be-ashamed/>

</comment><comment><USER20> I used to refer to this in some conversations as the "permission to be less than perfect", but "permission to suck" is crisper :)

The "permission to suck" pill needs to be taken along with the "desire to improve" pill. In other words, pay less attention to your current state and more to the trajectory - i.e. where you're heading. If the heading is right, then it is only a matter of time before you reach an arbitrary benchmark of your choosing.

</comment><comment><USER21> I tell people all the time: you get good at things you enjoy sucking at.</comment>


Strange object found during a sonar survey of the sea floor - <AUTHOR> http://www.oceanexplorer.se/index.html

<comment><USER2> An attempt at a viral marketing campaign for the movie Battleship?

</comment><comment><USER3> I'm pretty sure I've seen this image circulate the Digg/Reddit cycle a few months ago referring to the Millennium Falcon being found.

<comment><USER4> That's amusing - I just logged on to post a comment to the effect of 'that thing looks just like the Millenium Falcon.'

</comment></comment><comment><USER5> It would be nice if they provided just a little more information. Maybe a sense of the scale?

<comment><USER6> Sense of scale? It's very obvious to me, the scale shows 300 m. I would say the object is 40x40 m.

</comment></comment><comment><USER7> Millenium Falcon. Perhaps it was hidden from George Lucas.

<comment><USER8> Strange that this comment is being down voted. This was also the first thing I thought of.

<comment><USER7> Because humor is not launching, I guess.

</comment></comment></comment><comment><USER9> and yet, all pages i see mention swedish tabloids reporting it. no mention of real news outlets.

didn't boter to click this as I've clicked several other links elsewhere.</comment>


StackOverflow Update: 560M Pageviews a Month, 25 Servers - <AUTHOR> http://highscalability.com/blog/2014/7/21/stackoverflow-update-560m-pageviews-a-month-25-servers-and-i.html <comment><USER4> StackOverflow shows just how powerful using a fast language can be. Compare the TechEmpower benchmarks for, say, Java vs Rails on a very simple JSON serialization benchmark (we can assume .Net would achieve comparable performance to the JVM):

[http://www.techempower.com/benchmarks/](http://www.techempower.com/benchmarks/)

The Java servers get around 900K requests/s on beefy hardware while Rails squeezes out 6K. That's a 150x difference! Any real application will be slower than this, and on cloud hardware you can expect to be a order of magnitude slower as well. You just don't have much headroom if you use languages like Ruby before you have to scale out. And once you scale out you have to worry about sys. admin. and all the problems of a distributed system.

It's a one off cost to learn an efficient language, but it pays returns forever.

<comment><USER5> > StackOverflow shows just how powerful using a fast language can be. Compare > the TechEmpower benchmarks [...] (we can assume .Net would achieve > comparable performance to the JVM)

Except those TechEmpower benchmarks show .NET is not nearly as fast as Java. I think StackExchange prove that the platform is NOT the most important: it's much more important to make performance a priority in all engineering decisions, to benchmark everything beforehand and develop new technology where there's no good standard solution available. (think Dapper, protobuf-net, StackExchange.Redis)

<comment><USER6> I'd like to add one caveat with respect to our C# tests: we did not run the tests on Windows in Round 9. With the help of contributors, we've recently revamped our benchmark toolset, and have yet to pull the Windows implementation of the toolset up to date. The C# data in Round 9 is exclusively from Mono on Linux.

To more accurately judge the request routing performance of C# tests on Windows, see Round 8 data. For example, see Round 8 limited to Java, C#, and Ruby tests on i7 hardware [1].

Other important notes on that data:

* The http-listener test is a low-level test implementation.

* The rack-jruby test is running on Torqbox (TorqueBox 4), which is based on Undertow and has very little Ruby code [2]. This test is mostly about Undertow/Torqbox, with a bit of JRuby.

Another interesting test is Fortunes on i7 in Round 8 [3]. Fortunes involves request routing, a database query, ORM translation of multiple rows, dynamic addition of rows, in-memory sorting, server-side templating, XSS countermeasures, and UTF-8 encoding. Here you will see Java frameworks at about 45K rps, ASP.NET on Windows at about 15K rps, and Ruby at about 2.5K rps.

[1] [http://www.techempower.com/benchmarks/#section=data-r8&hw=i7...](http://www.techempower.com/benchmarks/#section=data-r8&hw=i7&test=json&l=39e)

[2] [https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...](https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/rack/config.ru)

[3] [http://www.techempower.com/benchmarks/#section=data-r8&hw=i7...](http://www.techempower.com/benchmarks/#section=data-r8&hw=i7&test=fortune&l=39e)

</comment></comment></comment><comment><USER7> > The cost of inefficient code can be higher than you think. Efficient code > stretches hardware further, reduces power usage, makes code easier for > programmers to understand.

I'm curious what the reasoning is for "Efficient code ... makes code easier for programmers to understand". To my mind, efficient code (in this case, I assume coding to the hardware, as they mention elsewhere), has many benefits, but making it easier to understand is _not_ one of them. A useful comment by the code may help, but that's not a result of efficient coding, that's a result of good commenting practice.

<comment><USER8> Sometimes "efficient" and "easy to understand and be sure is correct" don't have to be mutually exclusive; see this example [1] of Java and Go.

Note that the code is autogenerated, so it should be equally efficient. The Go version also happens to be very simple and no different than most humans would write by hand (without trying very hard to optimize).

[1] [https://gist.github.com/shurcooL/9f94bbd021b4693cf584](https://gist.github.com/shurcooL/9f94bbd021b4693cf584)

<comment><USER9> The code samples in your gist aren't doing the same thing. The Java version decodes the UTF-8 stored in the protobuf into Java's native UTF-16 Strings on first access, while Go strings are native UTF-8 and so only a nil check is necessary.

Or are you saying that languages should always use UTF-8 natively? I would agree with you on that, but disagree that this proves your point that "efficient" and "easy to understand and verify correctness" aren't mutually exclusive. pb.getSomeId().charAt(1000) runs in constant time in Java (albeit failing to give correct data if getSomeId() contains codepoints higher than \uFFFF), but pb.GetSomeId()[1000] will give you garbage data in Go if your field contains non-ASCII text. To get a valid codepoint in Go, you'd need to do string([]rune(pb.GetSomeId())[1000]), which runs in linear time and omits the check for valid UTF-8 bytes.

<comment><USER8> That's a good point, it seems the Java code is checking if the entire string is valid UTF-8 while the Go version isn't. I wonder why the behavior of generated protobufs is different between the two.

</comment></comment></comment></comment><comment><USER10> Really hate the separate sites thing. I use a half dozen of them and they are computer related, so it is a pain in the ass always having to register and not being allowed to comment/answer at first, etc.. They have a few improvements now like importing your profile from other sites, but they shouldn't even have so many separate sites in the first place, just a tag or a category or something.

<comment><USER11> You get +100 rep bonus for linking your accounts, so you shouldn't have a problem being allowed to comment.

<comment><USER12> How? I can't seem to find this option in account admin panels.

<comment><USER11> It happens automatically when you use the same account on multiple SE sites.

</comment></comment></comment></comment><comment><USER13> "One problem is not many tests. Tests aren't needed because there's a great community... If users find any problems with it they report the bugs that they've found."

I'm often surprised at the paucity of test-coverage in relatively large companies.

[http://nerds.airbnb.com/testing-at-airbnb/](http://nerds.airbnb.com/testing- at-airbnb/)

<comment><USER14> I think we're going trough a thesis/anti-thesis cycle on tests - in the beginning, there was militant testing, 100% coverage, testing getters and setters etc (as well as more complex stuff, obviously). Then some people started coming around to the idea that there are actually large swathes of code that is simple enough that testing doesn't actually add much value especially compared to the effort of writing them, then that probably got a bit out of hand (to what you're referring to). Maybe the pendulum will swing back and we'll find a good heuristic for just how much testing is the right amount that isn't all or nothing?

<comment><USER15> It really also does depend on the language, or rather compiler. Specifically, tests are far more useful when static analysis doesn't exist.

</comment></comment></comment><comment><USER16> > With their SQL Servers loaded with 384 GB of RAM and 2TB of SSD, AWS would > cost a fortune.

I have next to zero experience with server administration, but 384GB seems like a lot to me. Is that common for production servers for popular web services? Do you need a customized OS to address that much memory? Seems like you'd really need to beef up the cache hierarchy make 0.38TB of RAM fast.

<comment><USER17> I don't believe that much RAM is uncommon for large scale database servers. 384GB RAM is only about $5000 from Dell. They also have a new server model coming out that supports up to 6TB of RAM [0].

[0] [http://www.dell.com/us/business/p/poweredge-r920/pd](http://www.dell.com/us/business/p/poweredge-r920/pd)

<comment><USER18> How common is it to buy _one_ extremely powerful database server? Wouldn't you need the high availability properties of something like a Galera cluster? Or are most people accepting the single point of failure at huge database server?

(I honestly don't know, it's just surprising.)

<comment><USER19> While you do want HA you also want to keep things simple. You get great HA with just one backup machine. The times you lose two machines are so few that you might crash the site with software failures far more often anyway.

Databases have inherent locking problems that takes more resources to resolve with more machines, more so in some database systems than others. When scaling you often hit a point where you get down to macro logistics, so imagine a car highway: You can get more throughput by adding more lanes. But what that all cars are going to merge to one lane at some point in the middle of the trip (because of a tunnel, bridge or something: it's impossible to have more than one lane at that point)? Now you won't get any throughput benefits of the multiple lanes after all! Just more latency because you get queues up until the single lane and because of the queues all cars are going slow and need to accelerate on the single lane, so the average speed is low too. You are also having too much resources after the single lane because you can never fill all those lanes. It might be better to have one beefed-up single-lane road all the way that people can go fast on. Basically: Remove locks and you get better overall performance.

Yes, this is on the expense of HA. Yes, the costs of scaling up grows asymptotically faster than the costs of scaling out. So this is definitely a trade-off in some sense.

</comment></comment></comment></comment><comment><USER20> I've recently started using their micro-ORM Dapper, and I like it a lot. I get the performance of hand-coded SQL, but without the tedious mapping from SqlDataReader to my entity.

<comment><USER21> Dapper is great - I promptly dumped NHibernate and Entity Framework for it. It's the only ORM I've found that makes it trivially easy (as it should be) to use stored procedures and views in queries.

<comment><USER22> +1 for Dapper - I'm a big fan. It is very fast and we use it to serve our live (mostly read-only) traffic, while using something like Entity Framework for the backend Administrative UI, which has a lot of inserts and updates.

<comment><USER20> There's a couple of user-contrib projects that do similar things for inserts & updates. Dapper Extensions is the one I'm using.

[https://github.com/tmsmith/Dapper- Extensions](https://github.com/tmsmith/Dapper-Extensions)

<comment><USER21> I wrote a Dapper extension for working with SQL Server's geospatial queries and types a couple of years ago - have they added anything like that yet? Otherwise I'd be happy to add it.

<comment> Nick-Craver Marc Gravell added in the ability to put in pretty much any custom type without adding any dependency weight about a month ago. You can see the commit here: [https://github.com/StackExchange/dapper-dot- net/commit/e26ee...](https://github.com/StackExchange/dapper-dot- net/commit/e26ee0abe5bdff561ff59fded87cabb9f5d983a1)

Look towards the end at the tests for example of how to hook up a custom type (it's pretty simple).

<comment><USER21> Cool - the one I wrote was years ago. Looks like it's not necessary any more.

</comment></comment></comment></comment></comment></comment></comment><comment><USER23> So this is what, a 2000 request/sec peak? Over 11 servers, that's like 200 requests/sec peak per frontend?

The problem with scale-up is if you actually have to get a few times larger, it becomes super expensive. But fortunately hardware is increasing so much that you can probably just get away with it now. There's probably a crossover point we're rapidly approaching where even global-scale sites can just do all their transactions in RAM and keep it there (replicated). I know that's what VoltDB is counting on.

<comment> Nick-Craver Peak is more like 2600-3000 requests/sec on most weekdays. Remember that programming, being a profession, means our weekdays are significantly busier than weekends (as you can see here: [https://www.quantcast.com/p-c1rF4kxgLUzNc](https://www.quantcast.com/p-c1rF4kxgLUzNc)).

It's almost all over 9 servers, because 10 and 11 are only for meta.stackexchange.com, meta.stackoverflow.com, and the development tier. Those servers also run around 10-20% CPU which means we have quite a bit of headroom available. Here's a screenshot of our dashboard taken just now: [http://i.stack.imgur.com/HPdtl.png](http://i.stack.imgur.com/HPdtl.png) We can currently handle the full load of all sites (including Stack Overflow) on 2 servers...not 1 though, that ends badly with thread exhaustion.

We could add web servers pretty cheaply; these servers are approaching 4 years old and weren't even close to top-of-the-line back them. Even current generation replacements would be several times more powerful, if we needed to go that route.

Honestly the only scale-up problem we have is SSD space on the SQL boxes due to the growth pattern of reliability vs. space in the non-consumer space. By that I mean drives that have capacitors for power loss and such. I actually just wrote a lengthy email about what we're planning for storage on one of our SQL clusters...perhaps I should echo it verbatim as a blog post? I'm not sure how many people care about that sort of stuff outside our teams.

Nick Craver - Stack Exchange Sysadmin & Developer

<comment><USER24> >I actually just wrote a lengthy email about what we're planning for storage on one of our SQL clusters...perhaps I should echo it verbatim as a blog post? I'm not sure how many people care about that sort of stuff outside our teams.

I'm sure some DBAs and devs here would find it interesting.

<comment><USER25> This is definitely the case. Every writeup from those in the trenches I read and share with coworkers.

</comment></comment></comment></comment><comment><USER26> Makes me wonder why smaller website teams need dozens of engineers to keep their infinitesimally smaller app running.

<comment><USER27> I personally think it's due to overly complex applications and systems with many moving (and breaking) parts.

Also StackOverflow often practices "scale-up" instead of "scale-out".

<comment><USER28> More specifically, I think it's due to long feature lists and multiple problem domains. I'm also thinking about in-house line-of-business software.

</comment></comment></comment><comment><USER29> Extremely interesting read, especially because it goes against many fashionable engineering practices. They see (correctly) that abstractions, DI have their own compromises. Also, they're not religious (about technology), that's also quite rare.

</comment><comment><USER30> "SO goes to great lengths to reduce garbage collection costs, skipping practices like TDD, avoiding layers of abstraction, and using static methods."

I don't understand this at all. What does TDD have to do with reducing garbage collection?

<comment><USER31> My guess is that they feel that the layers of indirection and abstraction often needed to make TDD work result in an object creation pattern that results in heavy GC load during normal operation. The references to "using static methods" is probably related to this.

ps. That's my guess, but I'd encourage you to post your question to the meta site for SO.

<comment><USER32> IME that always happens when you try to perform tdd in combination with Javaesque encapsulation. The good solution to the problem is to not be so afraid of classes seeing each others internals. The bad solution is to add, factory patterns, dependency injectors and other useless layers just to try and keep your design both well encapsulated and testable.

<comment><USER33> You do realize that most efficient DI framework only inject the dependency once.

Separating Controller, Repository, and Services are good practices as well and let's be honest, we're looking at 3 methods layer at most.

Here's what happened in Java:

1\. When you deploy your WAR, the DI will inject necessary component _once_ (and these components only instantiated _once_ for the whole web-app so there you go, Singleton without hardcoding).

2\. A request comes in, gets processed by the pumped-up Servlet (already DI- ed). If the servlet has not been instantiated, it will be instantiated _once_ and the instance of the Servlet is kept in memory.

3\. Another request comes in, gets processed by the same pumped-up Servlet that is already instantiated and has already been injected with the component (no more injection, no more instantiation, no more Object to create...)

So I've got to ask this question: what GC problem we're trying to solve here?

Some of the static methods are understandable but if Component A requires Component B and both of them have already been instantiated _once_ and have been properly wired up together, we have 2 Objects for the lifetime of the application.

I'd pay for a wee bit extra hardware for the cost of maintainable code ;)

Discipline (and knowledge of object graph) definitely help to reach to that point.

<comment><USER32> You probably meant to respond Marco's comment? And afaik Stackoverflow is written in ASP.NET, not Java.

<comment><USER33> C# or Java, the whole request pipeline processing should be more or less the same. Unless one platform does things less efficient than the other.

Rails and Django do things differently as to my knowledge they do it by spawning processes instead of threads. There are app-server for Rails or Django that may use threads for efficiency/performance reason but I am under the impression the whole LAMP stack is still 1 request 1 process (even though they re-use those processes from a pool of already allocated processes).

</comment></comment></comment></comment></comment></comment><comment><USER34> What I'd like to see is how much cache hit they add on each level of cache. I remember some presentation of the facebook images stack where less than 10% (if I remember well) of the requests actually hit the disks; it would be interesting to see the patterns for the whole SE galaxy.

<comment><USER35> As someone mentioned above, SQL Server hits memory first, and as they have 384GB of memory, most of the requests would sit there. That's just on the DB server. In the linked article it all authenticated requests hit the DB, with anon users getting a cached copy.

</comment></comment><comment><USER36> Relatively new stack dev here. I came in on the other side of the fence of a lot of these technologies (bread and butter is Python in Flask with Mongo on Heroku on a mac) but since I started here, I've been constantly and pleasantly surprised by how performant everything here has been despite my biases. It's mighty fun.

</comment><comment><USER37> Stack Overflow is the example I'm forever using when arguing against premature scale-out. For non-trivial applications scale-out has substantial complexity costs attached to it, and the overwhelming majority of applications will _never truly need it_.

It's frustrating to see time wasted obsessing over trying to maintain eventual consistency (or chasing bugs when you failed to do so) on systems that could quite happily run on a single, not that beefy machine.

<comment><USER38> > __ _For non-trivial applications scale-out has substantial complexity costs > attached to it_ __

Forgie me if I am misunderstanding you - but non-trivial applications can actually require scale out.

From my perspective, StackExchange is not techinically that complex. They have built a very efficient, cost-effective and performant stack for their singular application and that works very well for them, but the complexity of their forum is not an extraordinarily complex problem.

<comment><USER37> By nontrivial I mean 'application which, in your document database, requires multi-document updates to perform some individual logical operations'. This is a rather low bar :-).

The fact is, the vast majority of projects that programmers are working on are less computationally complex than stack overflow. That's not to say that forum software is all that complex, more that most problems are pretty simple. Of course there are real reasons to use scale out - I simply advocate thinking hard about whether your problem will ever truly need it before taking the substantial complexity hit of coding for it.

<comment><USER37> Er, I don't seem to be able to edit, but I guess I should specify that I'm really talking about scale-out of writes here. Read scalability is an obviously easier problem..

</comment></comment></comment></comment><comment><USER39> > AWS would cost a fortune.

Isn't that comparing apples to oranges? Taking any application built for large servers and putting the same application onto a cloud-based architecture will be more expensive.

Cloud-based architecture requires ground up differences in how the application is built. Now whether or not it is better to use a cloud-based approach or traditional bare metal is highly subjective and isn't my point.

<comment><USER40> _> 'Isn't that comparing apples to oranges?'_

On another site, in another context, it probably would be, but here it's really presented as a contrast of scale-up versus scale-out - something the regular audience of highscalability will certainly grok.

In context...

 _' Stack Overflow still uses a scale-up strategy. No clouds in site. With their SQL Servers loaded with 384 GB of RAM and 2TB of SSD, AWS would cost a fortune. The cloud would also slow them down, making it harder to optimize and troubleshoot system issues. Plus, SO doesn't need a horizontal scaling strategy. Large peak loads, where scaling out makes sense, hasn't been a problem because they've been quite successful at sizing their system correctly.'_

It's an acknowledgement of their relatively unique strategy and the short list of caveats that make it possible.

</comment></comment><comment><USER41> I love that all the traffic is running through a single HAProxy box - <USER2> software load balancer (vs a hardware LB like F5).

And they've moved SSL termination to it.

That's a great quality product, and easy to setup.

Edit: I work I'm software that supports MySQL,PG and SqlServer. SqlServerr seems to be the most stable and consistent in performance - they're hard to kill! One of my few liked MS products :D

</comment><comment><USER42> This is the fascinating part to me, their SSD have not failed:

 _Failures have not been a problem, even with hundreds of intel 2.5 " SSDs in production, a single one hasn't failed yet. One or more spare parts are kept for each model, but multiple drive failure hasn't been a concern._

<comment> Nick-Craver Yep, still true. We lost one Intel 910 drive (PCIe SSD), and that was very abnormal - died so soon it was almost DOA. We hooked up directly with Intel for them to The replacement is still going strong as is another 910 we have.

All of those 2.5" Intels though, still trucking along! We're looking at some P3700s PCIe NVMe drives now, blog post coming about that.

</comment></comment><comment><USER43> One smart thing they are doing is putting different sites on different databases. It effectively as a kind of partitioning to allow for horizontal scaling if needed.

If nothing else, it keeps less data in each table, so queries should be faster due to smaller datasets.

Sounds like they know what they are doing.

<comment><USER44> beware that you can over-optimized this.

for example, do not put every single customer of an saas solution into a separate database.

<comment><USER45> I'm curious, why not? If your saas was small enough and had few enough customers with large data needs, separating them each out into a separate database seems like a viable solution.

<comment><USER44> yeah, if you have an experienced team, you can easily refactor your one- customer-per-db model into something that will handle 1,000 new customers signing up every month. given a good enough team, you can even do this while doing other sort-of important things like customer support, bug fixes, and new development.

however, inexperienced developers and teams are not good enough, or fast enough, to do that, and end up painting themselves into a corner when they have thousands of live customers to support and need to change their entire application architecture + database schema when their backup, replication, and housekeeping tasks choke on 1,000+ databases.

and oftentimes, one-customer-per-db also means one-instance-of-application- per-customer, another anti pattern to avoid.

we see this kind of stuff ALL the time. it happens a lot - people make terrible decisions and then are stuck with them 5 years down the line and are looking at a monumental cost to redo. not everyone is experience enough to work their way out of an awful situation like that.

</comment></comment></comment></comment><comment><USER46> It is fascinating to see how well its server it holding out. With Plenty of headroom to spare!.

With Higher Capacity DDR4, and Haswell or even Broadwell Xeon, and PCIe SSD getting cheaper and faster. It is not hard to imagine they could handle 3 - <USER3> times the load when they upgrade their server two years down the line.

But i would love to see Joel's take on it though. Since he is a Ruby guy now, and I dont think you could even achieve 20% of SO performance with RoR.

<comment><USER47> It would definitely be several orders of magnitude slower. That's the price you pay when you are using a language like ruby: it's very productive and fast to iterate in, but way less performant. At the end of the day, what you use depends on your needs.

<comment><USER46> Arh... I actually meant Jeff, not Joel.

Hardware is cheap, Programmers are expensive! But i am sure there is a line where this crosses over though.

<comment><USER48> The point of this article is that it costs more programmers to maintain a cloud than to maintain a single master database

</comment></comment></comment></comment><comment><USER49> Can anyone provide context on "Heavy usage of static classes and methods, for simplicity and better performance."

Is performance _really_ an issue here?

<comment><USER23> I find the opposition to static classes odd. Unless you need the data encapsulation of an object, why force people to create an object to call functions? I find too much C# code loves creating objects for no real purpose other than that's how it works.

For SO, if we guess they max out around 2000 req/sec, then if a bunch of objects are being allocated for each request, there could be added GC pressure. From my own experience developing much higher-scale managed code, allocating objects is a real pain and people will take a lot of effort to avoid it.

<comment><USER50> I think the real issue (although it doesn't seem to be an issue for SO) is testability. AFAIK, static classes cannot be mocked, and you are pretty much boxing yourself into only doing integration tests for everything. Not necessarily a bad way to go, just lots more setup per test.

<comment><USER51> Yep, that used to be one of the main reasons to have DI in Java. But now that everything (even static methods) can be mocked, that reason is no longer valid. Is this not possible in .NET?

</comment></comment></comment></comment><comment><USER52> Can someone help me understand what they mean by: "Garbage collection driven programming. SO goes to great lengths to reduce garbage collection costs, skipping practices like TDD, avoiding layers of abstraction, and using static methods. While extreme, the result is highly performing code."

<comment><USER53> I assume it means using C# more as a functional language than an OO language, so that most garbage collection is of short-lifetime objects and therefore cheaper to collect (generation 0).

<comment><USER54> I take that statement to mean C# as C. Functional languages are terrible for garbage collection. Static methods are great since they are static, loaded once and probably in-lineable by the compiler/JIT.

<comment><USER55> The usual wisdom in clr-land is that the generational garbage collector can cope just fine with a lot of small, short lived objects. It absolutely doesn't like big, long lived objects that reach G1, G2 or the large object heap and must be traversed for a full gc.

This means that normally, you want to avoid data structures that introduce a lot of objects trees, linked lists and stuff like that. In some circumstances you might lose some speed, but win big time when the full GC comes. Additionally, It may be worthwhile to consider avoiding heap allocations by organizing your data as value types instead of ordinary objects.

<comment> Nick-Craver It depends on your definitions of "a lot" is. It can scale pretty high by default, but when you're doing hundreds of millions of objects in a short window, you can actually measure pauses in the app domain while GC runs. These have a pretty decent impact on requests that hit up against that block.

The vast majority of programmers need not worry about this, but if you're putting millions of objects through a single app domain in a short time, then it's a concern you should be aware of. This applies to pretty much any managed language.

<comment><USER55> Are you sure that this is really because of Gen0 collections all by themselves, and not because the short cadence of collections leads to a lot of premature Gen1 promotions? I could imagine something like

       var a = InitGiantArray<Foo>();
       var b = InitGiantArray<Bar>();

could leave you with a lot of foo garbage in Gen1 or even Gen2, right were it really hurts. On the other hand I'd be surprised if something akin to

       do {
           Tuple.Create(new object(), new object());
       } while (true);

but with something useful thrown in would suffer much under GC pauses. Looks like I have to test it :I)

</comment></comment></comment></comment></comment></comment><comment><USER56> So funny because the most important application for programmers uses the complete opposite of almost every web development trend out there.

Also the SSDs are funny to me because I recently had someone who was supposed to be an expert say that SSDs were a fatal mistake with a high likelihood of massive failure.

</comment><comment><USER57> Soooo... a average of 200 pageviews a second - I suppose that doesn't sound as cool.

Or, to put it another way; 1 server per 8 pageviews per second

Does not seem that efficient when you put it like that. Even a resource hog like a wordpress install can eclipse that level.

*made correction- still not impressed

<comment><USER58> I think you're missing a 0 somewhere : 560000000 / (60 * 60 * 24 * 31) = 209

</comment></comment><comment><USER59> > Redis has 2 slaves, SQL has 2 replicas, tag engine has 3 nodes, elastic has > 3 nodes - any other service has high availability as well (and exists in > both data centers)

Does this mean the 2 Redis slaves, 2 SQL replicates etc. are in a secondary data center?

<comment> Nick-Craver We have 2 SQL Clusters. In the primary data center (New York) there is usually 1 master and 1 read-only replica in each cluster. There's also 1 read-only replica (async) in the DR data center (Oregon). If we're running in Oregon then the primary is there and both of the New York replicas are read-only and async.

For redis both data centers have 2 servers, the chain would look like this typically:

      ny-redis01
       - ny-redis02
         - or-redis01
           - or-redis02

Not everything is slaved between data centers (very temporary cache data we don't need to eat bandwidth by syncing, etc.) but the big items are so we don't come up without any shared cache in case of a hard down in the active data center. We can start without a cache if we need to, but it isn't very graceful.

Here's a screenshot of our redis dashboard just a moment ago (counts are low because we upgraded to 2.8.12 last week, which needed a restart): [http://i.stack.imgur.com/IgaBU.png](http://i.stack.imgur.com/IgaBU.png)

Nick Craver - Stack Exchange Sysadmin & Developer

<comment><USER59> Great info, Nick. Thanks for sharing.

</comment></comment></comment><comment><USER60> I would love to see comparing costs of their solution against cloud solutions as Azure.

They say that hardware is cheaper than programmers. Wouldn't that speak for the cloud solution (software based scaling instead of admins)?

<comment> Nick-Craver Hardware is cheaper than developers and efficient code. But you're only as fast as your slowest bottleneck and all the current cloud solutions have fundamental performance or capacity limits we run into.

Could you do it well if building for the cloud from day one? Yeah sure, I think so. Could you consistency render all your pages performing several up to date queries and cache fetches across that cloud network you don't control and getting sub 50ms render times? Well that's another matter, and unless you're talking about _substantially_ higher cost (at least 3-4x), the answer is no - it's still more economical for us to host our own servers.

And yes I realize that's a fun claim to make without detail. As soon as I get a free day (I think there's one scheduled in 2017 somewhere), I'll get with George to do another estimate of what AWS and Azure would cost and what bottlenecks they each have.

Nick Craver - Stack Exchange Sysadmin & Developer

</comment></comment><comment><USER61> So doing it right works. Nice :) I like these articles as it gives good examples/help supporting decisions when you dont want the crappy hack, but the do it right instead.

</comment><comment><USER62> I'm grateful to them them for being so transparent about how their systems work. It's rare to get this level of detail on how a large, successful site operates.

</comment><comment> UVB-76 Anyone else not quite expecting StackExchange to be using a Microsoft stack?

<comment><USER63> Joel worked at MS on office, Excel if I recall correctly. Jeff's blog is named for a feature from a book published by MS. I would be more surprised if it was MS free.

<comment><USER64> You mean to tell me Joel is partially responsible for the abomination known as Excel 2007 where I can't pull-drag the damn window from screen to screen (but all of the other Office 2007 products do!)

<comment><USER65> He seems to have left Microsoft in 1995, so no, not really.

</comment></comment></comment></comment><comment><USER66> I never ever ever never use SO's search feature: I land on the result through Google. So I guess they should give Google some credit for the claimed performance?

<comment><USER67> Google isn't serving Stack Exchange's pages to you. They published stats on pages served by their own servers. Google has nothing to do with it.

<comment><USER66> You miss the point: I never use the search feature of the site, I go to the answer/subject directly from Google's results. I do not think Google is running my keywords against SO's server in 'real-time' and returning the answers as results, Google returns indexed/cached copies. I am sure that many users behave like I do, no? They might as well keep a static copy of all pages in RAM and only update when new comments/answers are posted.

</comment></comment></comment><comment><USER68> I haven't been able to use SO since they shut off myopenid last year. I can't log in, so can't change my credentials ;-( Any ideas HN?

<comment><USER69> You can email us (as George said) or you can just punch your email address into the account recovery tool and get an automated email telling you what to click within the space of a few minutes: [http://stackoverflow.com/users/account- recovery](http://stackoverflow.com/users/account-recovery)

We... probably need to make that link a bit more obvious.

<comment><USER68> Thanks, that was easy. So now I just have a password instead of linking to another account @facebook or google+ I guess that works ;-)

</comment></comment></comment><comment><USER70> "Stack Overflow still uses Microsoft products. Microsoft infrastructure works and is cheap enough, so there's no compelling reason to change." <\- there's your problem :-)

<comment><USER71> 54th ranked site in the world with a small number of servers and apparently very maintainable and stable systems.

I would love that problem.

</comment></comment><comment><USER72> "with just 25 servers"

This really needs to stop. Go to stackexchange.com and you'll find that more than half of the HTTP requests are to cdn.sstatic.net.

Looking up the IP addresses for cdn.sstatic.net returned five entries for me, all owned by CloudFlare. None of the CloudFlare servers that they are using seem to be in that 25 count.

Sure, these are all for static assets, that isn't the point. There are way more than 25 servers being used to serve the StackOverflow sites.

<comment><USER73> Stack Exchange dont own their CDN. If you really start couting external servers, you would be adding DNS, possible client side proxy servers, client side network infrastructure, etc... these are things that OTHER people do, not SO.

<comment><USER74> I would say though, CDN doesn't fall in the same category as DNS. Serving static assets off other networks does take a huge load off the SE infrastructure and should probably be mentioned in the article.

<comment><USER75> Only in terms of bandwidth, though. The CPU load by processing is the interesting figure, not side noise by images/js/css.</comment></comment></comment></comment>


Ask HN: Strong vs. weak typing - <AUTHOR>

I'm working on a functional programming language and debating the issue of strong vs. weak typing. This is a dynamically typed, interpreted language, so it seems to me that some of the benefits of strong typing (i.e., catching type errors at compile time) are irrelevant.
Currently, for example, ("abc" + 5) will return "abc5", and ([1,2,3] + 4) will return [1,2,3,4]. The necessary conversions are done implicitly.
What I generally hear, however, is that strong typing is preferred. So, I'd like to know: what are some pros/cons of strong and weak typing? Is strong typing always the correct choice?
(For reference, the language I'm working on is Scotch: https://github.com/bendmorris/scotch) <comment><USER2> Strong typing makes it easier to reason about the correctness of your program. So even if you don't compile, you can reason about it (i.e. check your code). If that is important to you, strong might be the better choice. (Keep in mind it also carries an implementation cost)

If the point of the language is to enable rapid prototyping, you _want_ to be forgiving of errors. They're usually a simple oversight that doesn't significantly impact the outcome of the prototype.

I've oversimplified about a gazillion times. But ultimately, it boils down to "what is the point of your language". There is no absolute "better" for this argument.

<comment><AUTHOR> The goal is rapid prototyping. I'd also say that these types of operations can be considered features and not necessarily errors. While string + number is probably usually an error (who hasn't done that at least once), in Scotch, the + operator represents not only addition but list construction and concatenation - so Scotch's + is equal to +, ++, and : from Haskell.

</comment></comment><comment><USER3> I don't think there's a clear-cut winner - it's mostly choice. Strong typing generally allows more solid, less-breakable code; weak typing generally allows quick prototyping and small scripts to be done at a faster pace.

This is why I suggested a -strict flag (<https://github.com/bendmorris/scotch/issues/3>): if you decide that you're done prototyping, and would like to make what you're doing more explicit.

Another method could possibly be being able to specify in-code that it's strict, perhaps something along the lines of "#flags: strict <...>" for flags always applied to that file? (Yes, that's ugly. I'm talking about the idea, not that particular incarnation of the idea)

This would allow the best of both worlds, without trying to force them to play nice with each other in one file.

However, even while avoiding mixing them in one file, that still raises other issues. For instance, if non-strict file "a" imports strict file "b", is "a" strict? I'd vote not (this would allow the entire standard library to be strict for clarity, and not cause any issues elsewhere), but I'm curious what other people's opinions are.</comment>


TickStart Beta Signup (free) - <AUTHOR>

We (Squeejee) just launched the beta of our app TickStart. TickStart does time-tracking for small to medium sized consultancies or project-based businesses, with a focus on the business/reporting aspects of your team's productivity.
We'd like to invite the HN community to take it for a spin during our free beta.
You can signup at http://signup.tickstart.com
Thanks!
-Bradley Squeejee, Inc. <comment><USER2> Breathtaking UI design. Seriously impressed at how polished and smooth everything seems to be.

<comment><AUTHOR> Wow, thanks for the kind words! All the design credit goes to Squeejee co- founder Wynn Netherland. He's a genius! (@pengwynn on twitter)

</comment></comment><comment><USER3> Cool graphics, but seems a bit slow & clunky on FF3 ubuntu.

<comment><AUTHOR> Interesting... we haven't tested from a linux OS yet. I'll look into it. Thanks!</comment></comment>


For UK news.yc readers - See Paul Graham at Imperial College London on Dec 4th - <AUTHOR> http://www.imperialentrepreneurs.com/garage/paul-graham.php

<comment><USER2> >Paul Graham, arguably one of the most important figures in the Internet's history

Now that's just a little bit ridiculous, no matter how good his essays are.

<http://www.unc.edu/depts/jomc/academics/dri/pioneers2c.html>

<comment><USER3> He invented web apps...

<comment><USER2> Forgot about that. I guess that was pretty influential.

</comment></comment></comment><comment><USER4> As a london based Imperial student, YC news is turning into my number 1 events listing service.</comment>


Landsat 8 Data Now Available - <AUTHOR> http://landsat.usgs.gov/LDCM_Landsat8.php <comment><USER2> This is such an amazing dataset. I really like the cloudless map tiles that Mapbox are producing. <http://www.mapbox.com/blog/cloudless-atlas-with- landsat/>

</comment><comment><USER3> Does anyone know if there is a way to access the image archive in a more direct way than through one of the web interfaces ? FTP or WMS ?

<comment><USER4> Scenes are processed as you request them. Once they are processed they are available for immediate download by anyone.

Check out the FAQs on the landsat site: <http://landsat.usgs.gov/search_and_order_Landsat_scenes.php>

Also check out WELD. I don't think it has Landsat 8 data yet but it may meet your needs: <http://landsat.usgs.gov/WELD.php>

</comment></comment><comment><USER5> I wonder how this will affect the Maps data that Google/Apple/Microsoft has.

<comment><USER6> It probably won't, because its not very high resolution. Landsat is more about the spectrum coverage, 8 bands including infrared, and temporal coverage, a pass every 16 days, since 1984. This is useful for scientific studies.

Google/Apple/Microsoft buy their data from private satellites that at a much higher resolution and in plain visible RGB, and they only need a single cloud- free composite from any time, which is why some of the Google imagery doesn't have new buildings.

Google does store and provide Landsat and MODIS datasets in their Google(c)(tm) Maps Engine, and Earth Engine, for science related applications.

<comment><USER7>
        > Google/Apple/Microsoft buy their data from private satellites

The Google coverage people are most familiar with (i.e. major cities) is not from private satellites, but private aerial surveys.

<comment><USER8> That used to be the case, but these days it is mostly satellite data.

The newer civilian satellites (a lot of them are dual-purpose military/civilian) are capable of resolutions below 1m. ~2m resolution satellite imagery has been available for over a decade now (e.g. the IKONOS and recent SPOT satellites).

I think (?) most of Google's imagery currently comes from the GeoEye-1 satellite, which they partially funded.

<comment><USER8> Just to make to make what I'm saying more clear:

I'm certainly not disagreeing that the highest zoom levels in major cities are aerial photographs. The city-block level imagery available globally is from satellite, though.

For example, compare this (probably satellite, little building lean and ~1m resolution): <http://goo.gl/maps/mJadH>

To this (definitely aerial, lots of "lean" and ~10-20cm resolution): <http://goo.gl/maps/WYnvE>

</comment></comment></comment></comment></comment><comment><USER9> OK, so what kinds of things can you do with this data?

<comment><USER10> The frequency and depth of archive combined with non-visible spectrum records mean Landsat is great for monitoring changes over time over large areas. One of the best uses I"ve seen is monitoring forest clearing and burning in the Amazon.

<comment><USER11> Also, the long record of using the same spectrum bands within the Landsat program (going back to the 70s) simplifies measuring changes over time. This is a huge asset.</comment></comment></comment>