How We Went from 30 Servers to 2: Go - <AUTHOR> http://blog.iron.io/2013/03/how-we-went-from-30-servers-to-2-go.html <comment><USER3> It's not really "go" that makes the difference. it's how the runtimes and frameworks are used and/or made.

frameworks on top of frameworks, all being over engineered, with poor understanding of what the system actually does, result in super slow apps on top, that you generally go to aws to scale.

It's not the first time that I see people reducing a dozen servers that were "always maxed out" by a couple of servers "that can barely feel the load".

Yeah throwing hardware at the issues is fine'n all but we're been way over the limit too many times in too many directions.

Go provides a clear start/APIs/framework, and the language enforces good habits. Also, it doesnt have things like global interpreter locks.

The lesson? Stop using cool techs because there's blogs about them. UNDERSTAND the tech before using it.

<comment><USER4> I'm very sympathetic to your tech-on-tech-on-tech-on-tech hogwashery promulgated by open sourcers & vendors prosteltyzing their wonderful solutions and the other acculturation factors that permits developers approaches other than trying & seeing & exploring-

That said, you are wrong.

I'm not a Go user and I don't care for it, but Go is fundamentally different and better than most runtimes. Go has it's own green threading, which allows it to start new tasks using extremely small amounts of memory, and it can switch between tasks without the normal cost of context switching full on threads.

That is a huge difference. Node has some elements in play here, with it's event loop and callbacks, but really the only thing in the field that compares to Go that's in use is Erlang, and there's no frameworking on top of Erlang that wouldn't cause most PHP or nine tenths the Python developers to curl into a weeping ball at the Eldritch horrors they'd been exposed to: even ASM code is not nearly so effective, being clearly a low level somewhat harmless thing.

Being able to kick off small processes, that can sit around for a really long time, awaiting data- that's a model of computation we haven't done in decades, never with the mass popularity- there's a TLA for this kind of processing- _Communication Sequential Processes,_ and Go is it. It's entirely different from the procedural model, from the Python and Perl and PHP scripting, where concurrency is a carefully waded into thing- Go is a concurrent first runtime, about ongoing, enduring, concurrent processes, and that's huge, _HUGE_ I say, huge.

<comment><USER5> _> I'm very sympathetic to your tech-on-tech-on-tech-on-tech hogwashery promulgated by open sourcers & vendors prosteltyzing their wonderful solutions and the other acculturation factors that permits developers approaches other than trying & seeing & exploring-_

Are you studying for the SATs, or just trying to make your argument sound more compelling through the use of unnecessarily complex words?

<comment><USER4> It's how I feel, I'm so very very sorry it's tripped your acceptable tolerance levels. I'll bite my lip and not add "you prat" to the end. Thanks for your considered moderation, and I see your point.

<comment><USER5> My only point is that being obtuse is not the best way to get your point across. I assume your goal in expressing your opinion is to convince others, and to do so it helps to be clear, concise, and direct.

<comment><USER6> That's a very narrow assumption as to why someone would express their opinion. The OP might be sharing their experience (in what was a humorous way) purely to be sociable in this discussion...

Not everyone is out to convince others.

<comment><USER7> Righto, he could be here to make himself feel good by using words he doesn't usually get to.

<comment><USER4> Dare I call into question what you are here for? How do you feel about your contribution- attempting to heap more shame- plays for the whole audience? Is what you are doing of value?

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER8> Go's definitely picking up momentum. I know that Mozilla is shifting much of its services infrastructure to Go.

We're porting over our Arabic sentiment engine, currently in Python/Cython to Go. If you're dealing with simple data structures, going from Python to Go is almost a line-for-line port, but the performance benefits are, of course, massive. Our benchmarks show a 40x speed improvement so far.

Lastly, for anyone thinking about taking the plunge, use Go tip (from their source code repo) - don't use their "stable" releases. They fix bugs so fast you'll always want to be current.

<comment><USER9> Do you have a citation for Mozilla using Go? I would be a bit surprised given their(Mozilla) development of Rust.

<comment><USER10> Rust and Go are not similar or competing. They are both new, but aside from that the differences are huge. So if a project uses one, that doesn't mean the other was a possibility for that project too.

<comment><USER11> They're not similar at all, but both were designed to replace C++. That sort of means that they're competing for the use-case of "things I would write in C++ if it wasn't a god-awful nightmare to do so".

From there, though, you're right that they take very, very different approaches.

<comment><USER10> > They're not similar at all, but both were designed to replace C++.

In practice, the main use for Go appears to be like this story - <USER2> alternative to RoR, Node, Python/Django, etc. - and not a C++ alternative. Go and Rust may both start from the idea of doing C++ or something similar "right", but end up in very different places.

</comment></comment></comment></comment></comment><comment><USER12> I went briefly to through the comments, it seems that no-one is bothered with lack of details in this post, things are not logical at all, almost like it was written by PR agency in charge of promoting Go.

Except it is established business so I would assume things are true, but real motives are hidden. Go is obviously good language for some specific things, but it's not like ruby is pure trash, how come you did 'everything' in ruby, and then didn't know about eventmachine, but just had to rewrite things in Go.

And rewrite happened overnight, right?

A lot is missing from this story. I will definitely look more into Go, mostly because someone compared it in comments with TurboPascal, and I have fond memories of Borland tools, Pascal especially.

Go and Rails are so different that there is almost no point in comparing them.

<comment><USER13> > but real motives are hidden

Could you elaborate on this please? Are you referring to the OP's motives of publishing the article? Or of switching to Go?

> Go and Rails are so different that there is almost no point in comparing > them.

Except when one solves the same problem better than the other.

<comment><USER14> \- They did not say why they did not choose one of the other alternatives.

\- They did not say how they used the concurrent features of Go, only mentioned that they were there.

\- They did not say how long it took to rewrite.

\- They did not say what they changed in the API.

I'm not saying the story isn't true, but for a true story it lacks a lot. You could summarize the article with the title and you won't be missing much.

<comment><USER15> They didn't say _why_ Ruby was using a lot of CPU either. Some basic investigation into the root cause might've revealed some intractable problems that are tied to Ruby (or at least the MRI interpreter) like long GC pauses (something which can be mitigated by going to a different interpreter like JRuby versus a complete rewrite in a different language)

Instead, it's "OMG we're maxing out CPU, time to completely rewrite our app in a different language"

And how big is this app that it needs 30 servers? We probably serve an order of magnitude of traffic where I work than these guys do (just guessing) and don't need that many.

</comment></comment></comment></comment><comment><USER16> I was originally very excited about Go when I first learned about it. But then I got tired and frustrated quickly after having to listen to the other Gophers telling me that I don't need this or that feature because there is a better way to do it in Go.

Like.

I don't need exceptions because Go function can return multiple values.

I don't need a mocking framework like Mockito because Go has interfaces.

I don't need an interactive debugger because I can debug with command line using gdb.

I don't need named arguments because I can instantiate a struct and call my function with it. (Have you seen Ruby or Javascript code? Almost every function takes 'opts' as a single argument. Go is probably going down this path too.)

Then I learned about Scala. I'm not saying that Scala is better than Go. However, it has everything that I need. :)

<comment><USER17> Comments like this are weird. Go does not like exceptions. They've built a whole idiom around error returns. When you said, "I need exceptions", what did you expect them to say? "Oh, sorry, we forgot that, we'll get right on it"?

I also don't understand what you mean by "interactive debugger". What is gdb if not an interactive debugger? Gdb is my go-to debugger for Ruby as well.

<comment><USER16> Something like RubyMine IDE debugger or Firebug javascript debugger.

If you are a Ruby developer then I highly recommend trying out RubyMine. I promise that you will never debug Ruby code using gdb again!

<comment><USER18> I got all excited when I saw your comment and went and checked out the latest RubyMine video on YouTube, because having something like Firebug for Ruby would be awesome. But it looks just like what gdb gives you when being driven by a graphical frontend like DDD or Emacs. Actually it looks a lot less powerful than DDD. What am I missing?

</comment></comment></comment></comment><comment><USER19> For large projects, it'd probably make more sense to rewrite the critical code sections in C. rubyinline makes this really easy, and ruby C extensions aren't that bad either.

One time on a contracting gig I had some code that was interacting with cairo drawing and imagemagick. They didn't have a compatible raw image format at the time (one was RGBA and the other was ABGR, IIRC.) It's trivial to convert between the two in ruby, but it was taking 60 seconds per image. Once I switched that one loop to C, it took 0.02 seconds.

I've got nothing against Go, just suggesting an alternative that might make more sense in some situations.

<comment><USER20> This definitely does make a lot more sense than throwing out your whole app and rewriting it in many situations, but I'm always a little hinky about moving stuff into C just because it has so many ways to shoot yourself in the foot. You can obviously create bugs in any language (I know I certainly have!), but Go really does hit a nice spot between safety, productivity, and speed, so I can definitely see the attraction.

</comment></comment><comment><USER21> I've seen a lot of dumb Rails server configs and even dumber usages of Rails without any tuning at all.

With a couple weeks or a month of work I could shrink the hosting fees or resources consumed by a factor of 5 out of most Rails apps that are sitting up around 30 servers... and probably a factor of 10. Leaving aside whatever rookie or even intermediate mistakes were made in their Ruby code or their database, this post indicates a lack of understanding of what happened when their server fell over. Proper tuning of a deployment should not trigger a 100% failure mode like this.

These folks were itching to get off of Ruby for whatever reason... after all their roots were in Java. If your goal is to do a rewrite and learn a new language and gain some notoriety why waste time learning what you did wrong with Ruby or your server config?

<comment><USER22> Your inability to understand their clear description of a basic cascading failure mode under load speaks poorly of your actual knowledge and experience. Given that, I have to take everything else that you say with a large helping of salt.

<comment><USER23> Their description of the root problem was very superficial:

> At some threshold above 50%, our Rails servers would spike up to 100% CPU > usage and become unresponsive.

Yes, but why? What exactly were those processes doing? Why the sudden change at a particular threshold?

Their lack of detailed investigation into this makes their post useless to me -- I have no way of knowing (1) what specific aspect of Ruby's architecture makes it unfit for their problem?, or (2) is their application doing something stupid that causes the problem in first place?

<comment><USER22> If you read that sentence in context, your questions are answered. Here are the previous two sentences for context.

 _The bigger problem was dealing with big traffic spikes. When a big spike in traffic came in, it created a domino effect that would take take down our entire cluster_

Given the article to that point, it is clear what is happening. They are maintaining a steady state of X% (where X is above 50), then a big traffic spike comes along. That traffic load is not distributed equally (my guess is because requests are not created equal) and there is a hot spot. The hot spot fails, then increases the load on everything else. After this repeats a few times, there is insufficient capacity.

In other words at some steady state threshold above 50%, you wind up without sufficient capacity to handle traffic spikes. Nothing in this failure scenario is specific to Rails. It is a well-known failure mode for a cluster under load with a push based architecture (which http requests are).

The fact that you did not understand that description speaks poorly of your problem solving skills. Now they may well have been doing something trivial that is fixable to cause load to be less than it was. But you haven't convinced me that you're the person to be trusted to figure that out.

</comment></comment></comment></comment><comment><USER24> I'm getting the same experience here by entirely rewriting an old PHP web application [1] in Haskell using the Yesod web framework.

I've been using JMeter to benchmark both versions of the application. On a 10€/month dedicated server [2], the Haskell one was able to generate 220 dynamic pages per second [3] whereas the PHP one tops at 35 pages per second on a equivalent page.

Moreover, concurrence capabilities of Haskell are also pretty sweet : while I was benchmarking the web app using 2,000 concurrent connections, the application server was only using around 90MiB of RAM. I was not able to increase the number of concurrent connections as the client application I was using started to kill my quad core desktop, I suspect Haskell to be able to manage A LOT MORE concurrent connections as I didn't see any decrease in the throughput of the application as I was increasing the number of concurrent connections.

[1] <http://files.getwebb.org>

[2] <http://www.kimsufi.com/fr/>

[3] Static content has been ignored as modern servers like NGinx seem to be able to carry the static content (CSS, images, ...) at more than 5,000 req/sec on the same machine.

</comment><comment> grey-area Interesting to hear of Go being used in production. It'd be great to hear some more details on your setup when deploying the go processes - how are you managing failover, what's your load balancer, and how are you handling swapping out processes etc? Are you compiling on the server or local machines before deployment? Most other languages have lots of solutions on the deployment side now but Go is so new there isn't much info out there.

Also, as you are running an API presumably your app in Rails was pretty simple, did you have the impression things would be more complex in Go if you were writing an app with an extensive front end and UI and using sql? I'd miss all the view helpers etc available at present I think. Going from 30 servers to 2 certainly sounds like a huge improvement, so it was definitely worth it for you, are you thinking of writing any front-end apps in Go?

I've been playing around with Go recently and it is a fantastic language for someone coming from dynamic languages like Ruby. I particularly liked interfaces as a way to define a contract for implementations to follow, and the simple package system which encourages you to make your code modular.

<comment><AUTHOR> We actually have our own deployment tools for Go (and for Rails, our databases, etc). We built them before all the new hip options that are around today. We build on the target machines, although that's not a requirement since we all run the same architecture on our dev machines too (64 bit linux).

I'm not sure about writing a front end in Go, there's not a lot in terms of UI frameworks and Iron.io front end (HUD) is still in Rails so I can't really say much about it.

<comment> grey-area Interesting to know that you're building on target machines, I have yet to explore cross compilation, but it'd be great to be able to build before deploy.

Sorry, deployment was probably the wrong word - I was not so worried about getting the files up there (like you I have a simple home-grown solution for that), but more how you were handling process management, swapping out processes etc.

Perhaps with Go this is less of an issue because startup time for new processes is minimal, and you can simply kill one process, start another and not really miss any requests (unlike say Rails with startup times of 10 seconds or so for instances)? Did you find this wasn't a big issue in practice and something very simple works for you?

I'm currently playing with go but if considering it for client use would have to be sure that things like this were not an issue. Are you using any off the shelf tools for process management/load balancing or is it something that you have built yourself? Did you run into any problems?

</comment></comment></comment><comment><USER25> Ok, this bit has left me confused.

They were Java devs that liked ruby, they wrote applications in Ruby on Rails and the ruby apps were hitting limits so they immediately started looking at other languages.

But they don't mention the most obviously (to my mind) simple option.

JRuby

It is ruby (They like ruby). Most ruby apps can be run on JRuby with very very little changes (No need for a big rewrite) and it runs on the JVM with which they are familiar and it is very fast (and true multi-threading).

Maybe they did look into it but I would have thought that would be #1 on their "We tried this but discounted it" list.

<comment><USER26> In my experience, rewriting for JRuby is as much work as re-writing in a new language. Last I looked was a year or two ago, so it may have all changed, but it's not as simple as just moving your code over. Many really important gems don't work, or don't work properly.

<comment><USER27> The only gems that do not work are those which are not pure ruby. For the popular gems I think there have been rewrites. I haven't looked into JRuby but it has changed quite a bit in the last year from what I read

<comment><USER28> I don't think that's true. There are plenty of threadsafe gems with C extensions, and plenty of non-threadsafe pure ruby gems that mutate class/class-instance variable willy-nilly.

<comment><USER27> gems with C extensions are not supported <https://github.com/jruby/jruby/wiki/C-Extension-Alternatives>

I forgot about thread safety in pure ruby gems but that is not a huge problem - [https://github.com/jruby/jruby/wiki/Gems-known-not-to-be- thr...](https://github.com/jruby/jruby/wiki/Gems-known-not-to-be-threadsafe)

</comment></comment></comment></comment></comment><comment><USER29> > "We also weren't sure if we would be able hire top talent if we chose Go, > but we soon found out that we could get top talent because we chose Go."

I feel[1] that a smart/talented C/C++/anything developer can go from someone who has never seen or heard of golang to a proficient and productive Go developer in a matter of a few weeks, maybe even _days_, if not less.

That's how long it takes to go through the following materials (and fav some for later reference) and play with the language a bit.

<http://golang.org/ref/spec>

<http://www.youtube.com/watch?v=ytEkHepK08c>

[http://commandcenter.blogspot.com.au/2012/06/less-is- exponen...](http://commandcenter.blogspot.com.au/2012/06/less-is- exponentially-more.html)

<http://tour.golang.org/>

<http://golang.org/doc/install>

<http://www.youtube.com/watch?v=XCsL89YtqCs>

<http://golang.org/doc/code.html>

<http://golang.org/doc/effective_go.html>

<http://golang.org/pkg/> \- use as reference

<http://www.youtube.com/watch?v=f6kdp27TYZs>

<http://talks.golang.org/2012/splash.article>

<https://gobyexample.com/>

<http://golang-examples.tumblr.com/>

And a some more similar things that you can mostly get to from golang.org site. The beauty of how concise the language and even its website are, is that you can literally just go through everything there one thing after another.

[1] This is my personal opinion based on playing with go the last few weeks/months. I'd love to verify this theory. It's not yet the primary language in which I do things in (I use C++11 atm), but for all my side tasks[2] it proved to be indispensable. And I found it very easy to pick up. I can't wait until I start doing all my work in Go, that will be a true test of its productivity efficiency.

[2] <https://gist.github.com/shurcooL>

<comment><USER30> It's quite strange to me that people would identify as or look for a "[language] programmer". Sure, I happen to write more C++, Python, and C than anything else, but I've dabbled in just about everything and could reach comfortable proficiency in a matter of weeks. Most of programming and all of computer science is universal.

Any serious programmer should be a polyglot by default.

<comment><USER31> > Any serious programmer should be a polyglot by default.

It depends if you're going to spend years training someone or if you need an expert right now.

My experience is that it is impossible to maintain expert level skills in more than one or two language + library environments. You can remain familiar with other environments but you don't have the time to be an expert.

While I sometimes switch between C-family languages for different projects, it can take _months_ to get up-to-speed with the changes in a language and its environment since you last programmed in it. I'm talking about situations where I know the language well but the _environment_ has changed. Languages change a little but the libraries they use can change _dramatically_. And along with the change comes a whole body of implied knowledge about how to safely and effectively use it all and this impacts on how you use the language itself.

If you've literally never programmed in a language before, it can be a few _years_ before you know about all the eccentricities, before you understand why the language follows certain patterns, before you understand the risks with certain behaviors.

If you need an expert now, not in 12 months time, then they need to know both the language and the environment. If you can wait a couple months, then they still need to know the language.

<comment><USER32> _"It depends if you're going to spend years training someone or if you need an expert right now."_

I'm not going to claim that there are no legitimate reasons to hire people who are narrow experts in Blub (and only Blub), but I'm having a hard time thinking of any.

At an established company, you've got the luxury of time -- there's rarely a good reason to "need an expert right now" that isn't just a contractor. At a startup, where hiring the wrong person is a disaster, hiring an "expert right now" is like holding a loaded gun to your head. Ideally, you should be hiring "T" people -- lots of breadth, with lots of depth in at least one area.

A truly good programmer will pick up your language/framework in days, and be at full productivity in months, even from scratch. It's hard to do better than that.

<comment><USER11> Even at an established company, a really smart language newbie could write a bunch of ad-hoc code that does almost the same thing as well-understood libraries that everyone more experienced in the language uses. Even if his code was relatively high-quality, now you have a bunch of extra stuff to maintain, and everyone who interacts with it will have to learn this thing instead of just using the library everyone knows. I've done this in languages I'm unfamiliar with and will probably continue to do so.

<comment><USER32> Not Invented Here syndrome goes away with general experience. It doesn't come back every time you switch to a new language.

Just because I've never written Erlang doesn't mean that I will automatically try to write a random-number generator (say) the first time I need one in Erlang. I have enough experience to look for a library function first.

Empirically, NIH tends to be _more_ common in single-language developers, not less. People who place a lot of value on their "expertise" in Blub tend to do so because they're over-weighting the importance of their memory of the API details. When they don't automatically remember something, they leap to the conclusion that it doesn't exist. They're also typically a less-experienced cohort than people who have written in lots of different languages.

<comment><USER11> I wasn't talking about NIH syndrome, I was talking about "I don't know that a common library exists for this standard use-case so I'm going to write my own one-off because I have a job to do". I mean, you can google for libraries but sometimes you just don't find them and then find out a few weeks later what you should have used.

<comment><USER32> _I don't know that a common library exists for this standard use-case so I'm going to write my own one-off because I have a job to do._

That's called "laziness", and is even more likely than NIH syndrome to be mitigated by experience. Certainly, having lots of coding time with a single language doesn't make you less lazy, and having lots of experience with different languages doesn't make you lazier.

<comment><USER11> Well, I'm less lazy than most devs and have worked in lots of languages and did it just the other week -- maybe I'm just not that bright :)

<comment><USER33> Could you give a specific example as I am still thinking "random number generator" and want to balance it with your "standard idiom or library that everyone steeped in $lang knows "

Cheers

<comment><USER34> I dunno how well-known the things I didn't know about are, but an example of me doing it: <https://news.ycombinator.com/item?id=5113202>

<comment><USER33> I think there is a fine line between using a library and writin your own to understand better the domain

It's something to do with how critical the library functions are to you / your system. I would never write my own compression software, but I can see why people would just to learn about the trade offs.

<comment><USER34> Okay, but if I had known about the alternatives I would have used them, so I'm not sure how that's relevant.

<comment><USER33> philosophising really. I guess on the "actually helpful advice" level, I'm all out if the googles cant help.

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER35> Excellent success story with Go.

Love this quote... "We also weren't sure if we would be able hire top talent if we chose Go, but we soon found out that we could get top talent because we chose Go."

<comment><USER36> That's really obvious but overlooked. Developers just looking for their paycheck will only have learnt the established languages.

Developers who enjoy their profession will look to the future, excitement gleaming in their eyes.

I've heard of business developing in Haskell a good amount of success. I suspect, for the same reason.

<comment><USER37> It could also be said that developers who use established languages are excited to get things done rather than reinvent wheels continuously.

<comment><USER38> No, it can't. People don't use only Java because they "just want to get things done." They use only Java because that's all they learned, and all they need for their day job. If not, then they will invariably step over languages that are very interesting but obscure.

<comment><USER37> I'll give an anecdote. I've spent the last 7 years using Ada, a language that is /both/ interesting and obscure, almost exclusively. It's really a fantastic language that is frequently neglected much like poor Go, but with more merit. Anyway, in preparing the launch of my company's first iteration of its podcast network (<http://76streetnetwork.com>), I absolutely wanted to use Ada, but the allure of the utterly fantastic tool support and breadth of libraries available for Java had me up and running in a day or two despite having never written anymore than basic hello world-like crap with it in the past. There's really nothing crazy about Java, you're right, but I live for coding and I have a deep interest in quality software development, but I guess I'm just looking for a paycheck for this side project that will never make me any money! Ironically, Ada is what pays my bills.

<comment><USER38> Don't get me wrong, I'm not saying you shouldn't use Java--just that "I've only ever used Java" is a pretty good indicator of somebody not being genuinely interested in software development, and different ways to do things. I'd take somebody who learned Ada for fun over somebody who learned Java in school (and did nothing else) for a Java job any day.

</comment></comment></comment></comment></comment></comment><comment><USER39> "the entire process started up with only a few hundred KB's of memory (on startup)"

There is no way this could be true: last time I checked, a bare minimal Go HTTP server requires at least 2.8MB of memory on 64-bit machine. Are you using the default net/http library?

<comment><USER40> Yeah, my HTTP server made in Go usually starts up at around 6MB or so.

</comment></comment><comment><USER41> This story sounds mighty familiar. I realize the topic is about one language over another, and most of the comments here seem to be focusing on that, but there's another part which I find fascinating: the cascading failure mode when they'd lose a web head.

"This would in turn cause the load balancer to think it failed and take it out of the pool, thereby applying the load that the unresponsive server would have been handling to the remaining servers. And since the remaining servers are now handling the load of the lost server plus the spike, inevitably a second server would go down, the load balancer would take it out of the pool and so on."

It was 8 years ago, and the customer was running PHP, not Ruby, but otherwise it's the same basic story that lead to the creation of "Surge Protector", or its actual name, "Suicide Pact". <http://rachelbythebay.com/w/2011/06/28/sp/>

<comment><USER42> Keeping in theme with the suicide pact, STONITH, the other kind of pact for high-availability machines comes to mind - Shoot The Other Node In The Head.

</comment></comment><comment><USER43> Why did go "win" over erlang, scala, and clojure? Because it's becoming trendy and "fun?"

And, why are "fun" and "joy" terms ruby bloggers use for languages? Is this code for "easy" and "familiar?" As in, "this language is easy to learn.. It does not require us to learn difficult but mind-expanding concepts to become proficient."

Replacing ruby/rails with something (just about anything!) results in far fewer servers. Isn't this obvious to all? Is this really news to the average HN reader?

If that's an I/O bound API server, bet those two servers could be brought to one in a language/runtime that's more productive but a little less "fun" to learn.

<comment><USER4> One could adopt some kind of event-based system in scala or clojure, but Erlang and Go are alike in being the lone runtimes where running millions and millions of very small messaging processes is AOK no problem for the runtime, and not something one has to work really hard for.

Fun and joy are terms applicable here because the alternative is Erlang. Zing!

Go is rather ideal for these guys use case: if it wasn't an easy win, given what a snug fit their use case is, there would've had to have been red flags abound, as this really is a nearly idealized work load for Go's use: a hell of a lot of processes which sit around doing nothing, where one occasionally gets a message and forwards it along. Perfect Go story, as Go's lightweight processes (goroutines) are ideal for this kind of _Communicating Sequential Processes_ routing workload.

<comment><USER44> Haskell's runtime makes for cheaper threads than Erlang. Possibly cheaper than Go's, too.

<comment><USER4> Are they also in Haskell as in Erlang pre-emptable by the runtime?

There's certainly some cost in this design decision, more context that has to be swapped in and out and more state kept, but it's important capability to allow blind design & use- think Node, where everyone has to keep re-iterating how important it is to keep yielding to the event loop, to not do a lot of CPU work in a handler: irrelevant in Erlang world- in spite of the threading model not being OS threads, Erlang is happy to drop your lightweight thread on the fly when it sees fit.

<comment><USER45> > irrelevant in Erlang world

Exactly. That is a subtle distinction but for cases where responsiveness and low latency is important that is key. Another thing Erlang has is isolation of process heaps. If one process crashes, it won't affect others. No shared data structures between processes. It all goes to fault tolerance but also a major win for a completely concurrent garbage collection.

<comment><USER46> > If one process crashes, it won't affect others.

Unless they're linking or monitoring one another, an other important property of tolerant and distributed system.

<comment><USER45> But that is on purpose. Sometimes you do want that. In other words they are explicitly set up to monitor/link each other.

<comment><USER46> > But that is on purpose.

Indeed, but I was pointing it out because it's important to be able to do it.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER47> This could easily be rephrased as "How We Went from 30 Servers to 2: Static Typing and Compiled Code".

</comment><comment><USER48> I rewrote my DNS checking tool (<http://www.dnsinspect.com/>) in Go and I saw huge differences in resource usage (previously it was implemented with Ruby on Rails + EventMachine), my memory usage went down from 128MB per background worker to a few KB.

Now I'm able to run hundreds to thousands of concurrent reports using a small VPS, the Go application is using 36MB of RAM (24MB front end + 12MB for background workers). Go language is well suited for my particular case (many concurrent IO operations).

This was my first Go project, in a week I was comfortable with it, I had many alternatives but I really liked the simplicity of Go, fast compilation and easy of deployment. Because I've missed some pieces from the Ruby's world I've combined Jekyll (Compass, HAML SASS, RedCarpet, etc) with Go. :)

<comment><USER49> Do you also used the default net/http support libraries?

<comment><USER48> Yes, I'm using net/http.

</comment></comment></comment><comment><USER50> I'm missing the connection between poor API performance under Rails and the decision to do a ground-up rewrite. Was the initial performance problem due to an API problem or a platform problem? Was there any profiling of the poorly performing API server? I'd love to hear the story behind that analysis and how it impacted the tale told in TFA.

I ask all this since I've lost count of how many times I've participated in or witnessed an averted rewrite via a good dose of profiling and a few key bugfixes. I'll acknowledge that's not as fun as a clean-slate project, but _vast_ amounts of engineering time were saved.

</comment><comment><USER51> I agree with the 'Go is fun' (and easy to pick up) sentiment. I spent a few weeks recently writing an NTLM library for Go for a consulting client (I hope they'll let us open source it at some point). All the parsing and bit manipulation was _extremely_ straightforward due to the excellent standard libraries and I was able to quickly move from not knowing Go, to a fully implemented NTLMv1/v2 library!

</comment><comment><USER52> Another day, another Go PR post on HN.

What is new about porting code running in one of the slowest interpreters around to a compiled implementation of another language?

I was already porting Perl/TCL code to C++ back in 2002 with similar performance results.

Don't kids nowadays learn anything about performance in their CS degrees?

</comment><comment><USER53> It's an interesting tale, and I honestly have no bias toward Ruby or Go, but the cause/effect relationship is poorly illustrated (we switched languages). We all know runtime / performance / scalability is more complex than that. I'd like to hear where the bottleneck(s) were and how Go solved them.

</comment><comment><USER54> Last month I picked NodeJS and build couple of sites with it. This month, I want to pick up something new. I was hesitating between Python and Go. Can you answer couple of questions about Go from someone coming from NodeJS:

1- What is the state of the external Go libraries, especially DB (MySql), caching libraries (memcached), protocol libraries (Oauth). Are they stable 100%

2- How easy is logging and tracing in go?

Thanks.

<comment><USER55> Memcached's author (Brad Fitzpatrick) is a Go user, and wrote a memcache client for Go.

Go has a bunch of good stuff surrounding SQL. I haven't used MySQL, personally, so I can't comment on which MySQL driver is best, but I know YouTube uses Go for something related to MySQL, though I can't say for certain exactly what. <http://code.google.com/p/vitess>

The goauth2 (<http://code.google.com/p/goauth2>) library is written by Brad Fitzpatrick and Andrew Gerrand (two members of the Go team), and is stable.

Logging is great. Interfaces make it really, really flexible. I'm not sure what you mean by tracing. Stacktraces? Those are easy to retrieve: <http://golang.org/pkg/runtime/debug/#Stack>

<comment><USER17> Well, Go has a bunch of libraries for different SQL servers. I wouldn't go so far as to say it has lots of good SQL stuff; it's in approximately the same place as C is w.r.t. databases.

Having to interface with complicated SQL is a reason _not_ to use Go (but not an insurmountable one).

<comment><USER13> > it's in approximately the same place as C is w.r.t. databases.

I think the existence of a standardized interface like `database/sql` makes things strictly better.

An anecdote: I started making a new web application using MySQL. As soon as I realized the error of my ways (not because of the driver), I swapped out a single import from the MySQL driver to the PostgreSQL driver, and everything continued to work as it did before.

[EDIT] Anyone care to explain the down-votes?

<comment><USER56> The downvotes are likely because they're reading your MySQL-to-Postgres anecdote as the firing of a shot in an unrelated battle many love to hate, if not a mild troll.

<comment><USER13> Ah, right. My bad. The folly is still fresh, and there is still a bit of pent- up rage that I hadn't made this discovery sooner.

</comment></comment></comment></comment></comment></comment><comment><USER57> Has anyone had an experience of running websites from Go; or more specifically, how you handle none-HTML content?

I've been considering porting my CMS from mod_perl to Go, but I'm not sure how you work with the other files (CSS et al). I did read somewhere that you run Go from Apache but it's not recommended.

<comment><USER58> I've just started creating toy projects in Go. I put these Go projects (and Apache, which runs older PHP projects) behind nginx.

For hosting CSS, you either generate it programmatically and send the right Header, or you can use http.FileServe from the standard library [1]. (Surely other approaches are possible, but those are the two I've played with so far.)

[1] <https://code.google.com/p/go-wiki/wiki/HttpStaticFiles>

<comment><USER57> Thanks for that.

The issue I have is that I would prefer to keep Apache around if I can (I have a few sites - some of which I host for friends). While my web server does have a few IPs attached, I'd rather not have to buy more IPs just to separate Apache from Go.

And to be perfectly honest, I do quite like Apache. (each to their own I know, but I've had little reason to complain about it).

<comment><USER13> I use Apache too (it's all I've ever known), and run a few Go web apps. Everyone lives quite peacefully together. With Apache, I just set up a simple reverse proxy.

Here's an example of one I use to run my own `godoc` server:

        <VirtualHost *:80>
            ServerAdmin firstname.lastname@example.com
            ServerName godoc.burntsushi.net
            ProxyPreserveHost On
            ProxyPass / http://burntsushi.net:8080/
            ProxyPassReverse / http://burntsushi.net:8080/
        </VirtualHost>

<comment><USER57> Nice idea.

I might try that before dipping my toes into Nginx

<comment><USER13> Yeah, it's a nice holdover. I keep hearing great things about Nginx though. Someday I'll have time to check it out :-)

</comment></comment></comment></comment></comment></comment><comment><USER59> It's funny that they argue that they dumped the JVM derivatives like Scala, but wouldn't exactly tell why, nor support their argument with any sort of data/logs.

I am genuinely curious to know why they chose Go over Scala. If it was the syntax, etc. I can partially agree because it's one of scala's weak points, but then they pitch the main reason citing performance, so I'm genuinely curious to know.

<comment><USER60> He mentions JVM memory usage.

<comment><USER59> I think he edited it after me and many other posted similar comments, coz I didn't notice it the first time...

</comment></comment></comment><comment><USER61> "We decided to rewrite the API" and now it is over 300x faster. Go had nothing to do with this, it was obviously some boneheaded original design. Probably something like fork+exec on a Ruby VM on each request to enter a chroot.

I know HN has a hard-on for Go but come on.

<comment><USER62> Are you the same asshole making throwaways to shit on all of the Go postings or are there actually more than one of you going around making assumptions, accusations and spreading falsehoods about Go.

I can't get over how pathetic it is to make a throwaway for that comment, I'm more embarrassed five people validated your comment and attitude.

</comment></comment><comment><USER63> Alternatively...

why we needed 30 servers: Rails

</comment><comment><USER64> I've read this domino-effect on server clusters on several HN postmortems and I've seen various flavors of this on our own web servers. I'd like to think there's a simple configuration in most servers that prevent 100% cpu utilization from taking place and preventing the server from telling its cluster that it's still alive. Anyone have any experience with this?

<comment><USER57> It's not really possible without adding resources (ie a new server).

The problem is, once your servers get saturated, requests queue up, processing slows down and users start to hit refresh which artificially and exponentially increases traffic. Then to compound things, servers buckle, which means you lose a resource right when you needed it the most.

A domino effect is one analogy, another might be how a small hole in a damn can buckle and blow open from the force of the water pushing through.

<comment><USER23> > It's not really possible without adding resources (ie a new server).

That's not quite true. With the right architecture, performance will degrade gracefully. You'll get slower and slower responses as the load goes up, and eventually start dropping requests, but the servers will not die and there will be no cascading failure.

One way to achieve this is to make sure the queuing happens at the load balancer, and no large queues are allowed to build up in the individual application servers.

<comment><USER57> Well yeah. You can do all sorts of tricks from TCP/IP hacks to streamline HTTP requests through to disabling queuing entirely. But my point is you cannot entirely prevent your site from saturation without adding extra servers to your web farm (and more so, that _siavosh_ 's method of taking servers out of service has the inverse effect of what he was trying to achieve).

Thus all you can do is slow the escalation in the hope that the traffic peaks before your resources buckle.

The only method I'd found that is "guaranteed" to prevent such outages is the use of sorry pages (ie a static page stating "We're experiencing high volumes" which users are directed too if the dynamic page connections are maxed out). However even that is just essentially a prettier version of a page time out - and I mean this in terms of usability rather than technicality. ie the site is still unavailable, but you're killing the connection in a user friendly way rather than allowing connections to stack or just flat out disallowing "> _n_ " active TCP/IP connections.

</comment></comment></comment></comment><comment><USER65> Can you describe your workload a bit as well as any benchmarking you did to determine (and perhaps optimize) hotspots in the ruby code before you started the port?

Personally, I've been interested in moving to Go on a python-based project of mine. Thus far, I've avoided it because 1) the extra work required to self- implement a few third party libs I rely on and 2) I've been able to eek out sufficient performance using c extensions and cython.

<comment><AUTHOR> This first rewrite in Go was for the IronWorker API so all the operations are here: <http://dev.iron.io/worker/reference/api/>

There were less endpoints back then, but you can get the idea. The most heavily used operation being queuing up tasks/jobs.

</comment></comment><comment><USER66> Science and engineering from Bell Labs behind Go is what make this possible. It is not a "different language", it is carefully selected design decisions and ideas behind it.

It relies on a principle of being good-enough, not to stuff everything in, as a "feature sellers" and "buzzword shouters" used to do.

There are also lot of work of great minds behind Lisps or Erlang, and same principles in a foundation.

</comment><comment><USER67> "So then the decision came down to which language to use."

I find this tendency to constantly jackknife between talking about languages and talking about frameworks a little dizzying.

Shouldn't the decision have been what web stack to use? I'm sure you could get half way there with a bespoke Ruby web stack, maybe built on top of JRuby or with a sprinkling of C extensions. You could get the same poor scalability with some badly designed framework on top of Go [1].

There's a lack of meat in the post (hence why people are calling it out as PR). How do these features of Go make it easier to write applications or frameworks that cater to this particular scalability need? Could extra effort be put in up front to get something similar out of Ruby or any of the other languages, or will they always be sub-par? What is it about Go that makes it light up the rest of the stack in a way other languages don't?

Or have you just traded one trendy technology for another because it promises to be the magic bullet for your current itch?

[1] This isn't saying Rails is badly designed.

</comment><comment><USER68> Hey that's my photo of the GOpher! :)

[1] [http://www.flickr.com/photos/jianshen/8080852738/in/photostr...](http://www.flickr.com/photos/jianshen/8080852738/in/photostream)

<comment><AUTHOR> Hah, nice photo! Are you ok with that? Found it on google images.

<comment><USER68> It's cool. :)

</comment></comment></comment><comment><USER69> OK, I am sold on Go as my next language for web applications. Unfortunately, I can't throw away the existing PHP/MySQL code. Any suggestions on how to integrate Go and PHP into a single web server? Should I be using FastCGI for Apache to call the Go programs?

</comment><comment><USER70> Out of curiousity, I would be interested to know which version/patches of ruby you were using. With ruby I'm used to running out of memory on servers long before running out of CPU. Of course, most of our workloads are IO bound, so that may be the diffent.

<comment><AUTHOR> This was on Ruby 1.9, 2010 era. No patches. Memory was definitely a concern and probably contributed to some of it. Just to note though, after a machine was taken offline by the load balancer, it would generally come back to life.

<comment><USER71> Memory "probably contributed to it"? You don't even know _why_ the old version was slow but you want to credit golang with making it fast? Hype much? The lack of critical thinking here on Hacker News is astounding.

</comment></comment></comment><comment><USER72> What changed in the architecture? It is rather interesting that by just having a language change you could remove 28 servers from the system.

<comment><USER73> My guess is that removing 2-5 layers of abstraction can get you pretty far along.

<comment><USER74> We're writing most of the systems for our company in Go, and of the many reasons this ranked highly. Just removing the multiple unnecessary abstractions gets you a hell of a lot of simplicity and performance back.

<comment><USER73> I work in an area that requires a lot of high performance computing and there is a reason people in this area like code simplicity. Stacking layer upon layer of complex algorithms is a lot more difficult to reason about than having a simple and elegant system that's close to the metal.

There is a recent trend of building more and more complex systems, have less coupling and 'good oop'. That's all good and nice but simple code is:

      - more maintainable (a complex nest of objects & function calls all loosely coupled isn't)
      - a lot more readable
      - easier to optimize
      - better for your sanity

Some languages have build a culture of extreme complexity and are notable for building huge structures and systems to do simple tasks. Having a 50K lines of code dedicated to just inversion of control or O/R mapping makes a problem a lot more complex than it probably needs to be.

<comment><USER75> _"Having a 50K lines of code dedicated to just inversion of control or O/R mapping makes a problem a lot more complex than it probably needs to be."_

Exactly. That's the terrible stuff with all this "enterprisey" mindset: people are working on medium-sized codebase made of this special kind of hell that Java/C# + ORM ([N]hibernate) + XML + SQL is and these app often run into the 200K / 300 KLOC lines if not more. Yet what does these applications _really_ do? Actually not very much. Yet these programmers are sure to be working on super-advanced stuff because their _codebase_ is big.

When several companies reported a drop in LOC of 90% by switching to something else than Java then at one point you have to at least consider that maybe most of your Java codebase is hot air.

</comment></comment></comment></comment></comment><comment><USER76> It would be interesting to hear some of the reasons you went with Go vs Clojure (and also the other languages you mentioned)

</comment><comment><USER77> I know I'm late to the party, and no one will read this, but ...

        "... Java derivatives like Scala and Clojure ..."

Someone help me out here: In what way is Clojure a Javav derivative?

<comment><USER78> Perhaps in the sense that they are languages that target the JVM and that rely on interoperability with the Java standard library to bootstrap people making apps with them?

<comment><USER77> I can't see how it could mean anything else, but I find that, well, bizarre. Oh well, different point of view, I guess.

</comment></comment></comment><comment><USER79> > with Ruby.

Title should have been: "How we went from 30 servers to 2: Replacing Ruby"

</comment><comment><USER80> How about BDD, test-driven development and quality insurance? Does Go provide an ecosystem that supports agile refactorings that are common in lean startups?

You can say anything bad about the performance of Ruby and Rails etc. but rspec, cucumber, capybara, vcr, factorygirl are really important features to start from zero and reach a viable product.

<comment><USER16> Go doesn't have any mocking framework like Mockito/Java or Rspec mock. You will end up writing a ton of code yourself if you want to mock something simple.

<comment><USER62> Uh what? No, that couldn't possibly be less true. Have you ever heard of "interfaces" in Go? They're kind of an important feature.

<comment><USER16> That's what the other gophers say too. Java has interfaces too, but it also has Mockito that makes testing much easier. You should look into it. Mocking library will help reduce boiler code. You would know that if you had used one.

<comment><USER62> >You would know that if you had used one.

Cute.

<comment><USER16> My bad. :)

</comment></comment></comment></comment></comment></comment><comment><USER81> Uh I think I love it. Easy to learn language and awesome performance.

</comment><comment><USER82> I wonder how far they could have got going more of a bare-bones Sinatra app using JRuby. It's still not nearly as fast as Go obviously, but take away Rails and some of the more dynamic usages of Ruby on a faster VM and I'd venture to guess you might have been able to go from 30 servers to 10-15?

That's still not 2 mind you...

</comment><comment><USER83> There's simply no technology that makes you switch from 30 machines to 1 (you say you keep 2 just for reliance).

Obviously there must be other architectural changes in play here (I'm guessing RoRs thread-per-request vs some kind of event loop on Go?). Please be more specific.

<comment><USER84> Go is good at keeping all the cores on the machine pumping. Maybe they switched to machines with more cores.

</comment></comment><comment><USER85> Why Go over VisualBasic?

</comment><comment><USER86> There are a lot of things that can be said about the "average" programmer in such-and-such languages, what they do and don't do, what habits they have, etc. But I'm not interested in working with merely average people.

<comment><USER87> That's right. People like you and me work with the top 5% of programmers! It's because I'm smrt.

</comment></comment><comment><USER88> You have an error in Opera

'SyntaxHighlighter.config.bloggerMode = true;'

Undefined variable: SyntaxHighlighterError

</comment><comment><USER89> Go 1.1RC just around the corner in April. A lot of good stuff in Go 1.1RC.

</comment><comment><USER90> If you look for a Golang programmer today, it's more likely you'll find a _true programmer_ instead of a _me too_ programmer. So if you're a true programmer, learn Golang. ;)

</comment><comment><USER91> Now I know that the dynamic languages such as ruby and python are not known for performance, but the benchmarks I have seen with Go look like it isnt exactly the fastest language available either. After reading this article I am more apt to think that something is/was VERY wrong with their Ruby code than to think that Go is that much better. Maybe my lack of experience with either GO or Ruby is showing here, but Ruby cant be _that_ slow can it????

</comment><comment><USER4> p.s. come disrupt our business: with nearly no hardware, no costs, and no maintence burden you too can pass OMG huge amounts of message traffic.

this is one bold as fuck post.

</comment><comment><USER92> <http://www.dartlang.org/support/faq.html>

>> Q. How does Dart relate to Go?# Dart and Go are both language projects started at Google, but they are independent and have different goals. As a result, they make different choices, and the languages have very different natures, even while we all try to learn from each others' work.

Hmm, don't think I want to bet the farm on that.

<comment><USER62> Don't want to bet the farm on what, do you mean Dart or Go sticking around? Go is used extensively through Google. It powers a significant DB layer that powers YouTube. It serves up all of Google's static downloads and several employees hint at other significant projects they can't speak about.

I've only dabbled in Dart, I can't speak to how vital it is, currently it only seems viable when translated to JS or when using in a server-side VM, which is certainly an option if you want.

(sorry, I originally had written Google where I meant YouTube, see "Vitess" for more info)

</comment></comment><comment><USER93> _We decided to rewrite the API. This was an easy decision, clearly our Ruby on Rails API wasn't going to scale well and coming from many years of Java development and having written a bunch of things that handled tons of load with way less resources than what this Ruby on Rails setup could handle, I knew we could do a lot better_

</comment><comment><USER94> Was anyone else expecting this to be a boast from EA about how they cut their operating costs for the SimCity launch?!?

</comment><comment><USER95> man, you just made me worry very much now - we are making taxi dispatching api using ROR - and we will lunch it soon

<comment><USER96> Why worry? If you can handle the initial load, you can still optimize later on, rewriting some functions or parts of the system with something faster.

<comment><USER95> great

</comment></comment></comment><comment><USER16> I wish Go has named arguments like Python and Scala. Reading and debugging code could be so much simpler.

</comment><comment><USER97> Why Go over Erlang?

<comment><USER98> (Note: I'm a big Erlang supporter and 90% of my startup's code is written in Erlang)

Erlang is _not_ memory efficient (compared to Go? I don't have benchmarks to back this up) AND the learning curve for developers not exposed to functional programming (which is a lot) is quite a bit higher than it is in Go.

</comment></comment><comment><USER99> Hmm, wouldn't it be better to rewrite the ruby services to Go, the actual workers and not the entire API? I don't think the API HTTP requests where your main issues here...

</comment><comment><USER100> Would be great to hear more specifics of the rewrite. What did the API do, what pieces were built in RR, what got replaced in Go, etc.

At least as much as you're willing to share. :-)

</comment><comment><USER99> The problem was Ruby, not Rails. Did you give ever give Jruby a chance?

</comment><comment><USER101> Rails is not for API,why not use eventmachine?

</comment><comment><USER102> Why was Go chosen over NodeJS ?

<comment><AUTHOR> Mostly because I can't stand Javascript. It makes me cringe just thinking about it. Go is much nicer to work with.

<comment><USER103> Hey OP! I appreciate your sharing. Since you came from ruby and we're on the topic of the language itself, I'd appreciate your impression of how well Go supports collections. Is there or could one write something like <http://underscorejs.org/>? Can you do this kind of thing?

[1, 2, 3, 4, 5].reject {|i| i < 3}.map {|i| i + 9}

I did the go tutorial the other day and I became a little worried that one would not be able to do this kind of thing without a bunch of unwieldy declarations. I think that would be a showstopper for me.

Thanks for any insight you might have!

<comment><USER11> Go has first-class functions, so you could write a library do the same thing with, yes, a few more declarations around it. In your example, .map { |i| i + 9 } is .map( func(i int) int { return i + 9 }), or .map(myFuncDefinedElsewhere) in most real-world scenarios.

As a commenter upthread pointed out, Go and serverside Javascript are aimed at very different use cases and audiences.

<comment><USER55> I thought this would be a fun exercise, so I implemented the Go equivalent. Can anyone get it to fewer lines? It abandoned all pretenses of readability long ago.

<http://play.golang.org/p/M5VaeIgQ-q>

<comment><USER55> Updated to generalise a bit more. If this ever came up in code review, I'd probably facepalm pretty hard.

<http://play.golang.org/p/YUQSZgFx_b>

<comment><USER11> Ha. Yeah, the map() and reject() function defs are fine but the actual usage is a little eye-bleed inducing :)

<comment><USER55> So now that we've established Ruby is better at being Ruby than Go is, I wonder if there's an actual use case for this kind of thing, so I can demonstrate the "Go way" of handling it...

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER104> No Love for Lua and LuaJit.

</comment><comment><USER62> Man, what is it about Go that just brings out the pricks of HN?

There are over three throwaway accounts created exclusively for this thread to shit-post about Go, to steal an old term I have never, ever used on HN before.

<comment><USER105> I'm not exactly sure, I've noticed the same thing.

Go appeared a few years ago, quickly became pretty well-known, and continues to grow in popularity. Maybe it's seen as "threatening" to older projects like Ruby and Python? The most vitriolic comments always seem to come from Ruby, Python, and Rust partisans. Of course, I'm not sure why they're so threatened by it, they themselves explain in great depth how the lack of generics or exceptions will prevent anyone from writing any real code in it, or even trying to in the first place!

</comment></comment><comment><USER106> great stuff!!</comment>


Journey of a startup that's creating one of the best realtime app, ever - <AUTHOR> https://medium.com/@devezk/journey-of-a-startup-data-actions-people-actions-revisited-67beb3e45ac5

<comment><AUTHOR> Friends, two of us dreamt about creating a startup in 2012. We were young at that time - just 22 and 23 years of age, and we came from two different countries - NewZealand and India. We have built something that we believe is one of the most data intensive yet powerfully collaborative. Can't wait to share it with the community here. Do read and let me know how do you feel. Show us some love if you got a hint what we have built.

</comment><comment><USER2> We all know that ideas are important. As they say, ideas are like bulletproof. But we believe that ideas come in all forms and they are better when collective/collaborative. Can't wait to share it with the community here.</comment>


Show HN: UltraEd – A level editor/game engine for the Nintendo 64 - <AUTHOR> https://github.com/deadcast/UltraEd <comment><USER2> In case anyone else wants to know how it looks like:

[https://twitter.com/_deadcast/status/907485256827023360](https://twitter.com/_deadcast/status/907485256827023360)

<comment><AUTHOR> Haha thanks for providing that! :) I should eventually add some gifs/png showing the editor working.

<comment><USER3> Hey I was wondering if you knew of any sites that provide info about inner workings of n64 games like Ocarina of time or Rare games like Goldeneye? I have always wanted to learn more about these titles but I have not heard of any attempts to reverse engineer them.

<comment><AUTHOR> Yes! :) There's a great book called the Nintendo 64 Anthology that is a fantastic read to get in-depth history of the console. It also dives into how Rare created Golden Eye and some good info about Silicon Graphic's hardware development at the time.

For websites I would check out gamasutra. They great articles like this one [https://www.gamasutra.com/view/feature/131556/postmortem_ang...](https://www.gamasutra.com/view/feature/131556/postmortem_angel_studios_.php). I know they have some others like on Indiana Jones and Star Wars Ep 1: Racer.

</comment></comment></comment></comment><comment><USER4> Wow - only supports Windows XP, and a MediaFire link full of questionable executables? I hope fixing this is part of your "WIP"...

<comment><AUTHOR> These files are necessary to setup the dev environment if you want to compile. I use the link myself when setting up a new build machine.

If you don't trust the files that's perfectly fine. You could pretty easily, I believe, find them all from repositories you trust.

Also yep Win XP only! Keeping it authentic lol. :)

<comment><USER5> To clarify: The N64 SDK runs on XP.

<comment><USER6> I wonder if it'd be possible to get it running on Wine for a safer experience.

<comment><AUTHOR> I've tried but it doesn't seem possible. Wine and even DOS Box can't run the 16 bit SDK executables. I'd love to be able to run just the ROM build process in Wine. :)

<comment><USER4> What's so special about the ROM process that it can't be ported or rewritten in a more portable way?

<comment><AUTHOR> Because the 16 bit executables are closed source. :/ The special linker MILD.exe that organizes all of the RSP(Reality Signal Processor) microcodes and game data is the piece that needs to be reverse-engineered which is possible but currently beyond me. :)

<comment><USER4> I don't think it's beyond you, I just think you lack confidence. You can do it! I've been in your shoes before and it worked out.

<comment><AUTHOR> Aww thanks much! :D Yeah I just lack confidence I guess since reversing binaries has always seemed a bit intense. I need to finally buy a really good code reversal tool and then start my quest to reproduce! :) It would be great if me or someone else was successful since then the N64 could be more easily developed for on any OS!!

<comment><USER4> You probably don't even need to reverse anything. Just learn how other (open source) toolchains work, read N64 manuals and information from other reverse engineers, and work incrementally. It's probably not necessary to reverse this proprietary ROM you're stuck with.

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><AUTHOR> BTW here's a video of proof-of-concept ROM I built first to see if I could actually achieve building what I wanted on the console. :)

[https://www.youtube.com/watch?v=59txD0ZyBic](https://www.youtube.com/watch?v=59txD0ZyBic)

<comment><USER7> Out of curiosity, what cart are you using? And are you using a Nintendo or custom expansion pack?

<comment><AUTHOR> I'm using the 64drive cart from [http://64drive.retroactive.be/](http://64drive.retroactive.be/). I've owned both generations and they're very easy to develop with. There's a USB port on the side of the N64 cart that lets you load ROMs very easily to test rapidly.

</comment></comment></comment><comment><USER8> I like the concept behind this. How may I ask would you go about getting this onto the actual system, or is it just something to throw into Project64 or similar emulator?

<comment><AUTHOR> It's really easy actually to load on an emulator or on real hardware. The build process I'm working on now creates a production ready ROM that can be placed on an EverDrive 64 or 64 Drive cart. I always test on real hardware when developing the engine for the N64 part of the system. :)

<comment><USER8> Cool beans. This looks pretty exciting to me; it may not be the fanciest JS framework, but this is a genuinely fun, different project. I'll be watching this project for sure.

<comment><AUTHOR> Aww thanks so much! Glad you like it! :D Yeah this is definitely not useful haha but I'm learning more about 3d mathematics, rendering techniques, low- level systems, constructing tools, etc...

</comment></comment></comment></comment><comment><USER9> So like... I'd need to use this in a VM? I'm not going out of my way to install XP on my 8700k system. Just to see what it do, seems like a big ask.

<comment><USER5> What's hard about a VM? You can download an entire VM complete with OS and all relevant dev tools and have zero leakage of viruses or workspaces into your normal client. It's a great way to encapsulate your development for a specific task.

I use a Win10 VM in my Win7 machine for almost all of my dev work. It stays on an SSD and completely keeps all that outlook, Microsoft SQL, Visual Studio, IIS, NAV, CRM crap from destroying what is normally a fast machine for gaming. I also host it on a separate IP so I can RDP into it from my crappy work laptop (which becomes nothing more than a thin client) and have remote access (from say my phone wireless hotspot if wifi is unavailable) and I've got all my work stuff exactly as I left the house.

Recently, I put together a XP VM with Visual Studio 2003, 2005, and 2008 for supporting and upgrading an old .NET 1.1 project for a client. There's no way I was going to try and get that crap running on my 64-bit computer.

Last thing I want is FL Studio (and all my related audio tools), Photoshop (and all related graphics tools), and Visual Studio (across over a decade of versions) all on the same machine destroying each other. You can't even get multiple versions of Java to run on the same machine these days.. Ugh.

</comment></comment><comment><USER10> Does this run on ReactOS?

<comment><AUTHOR> Just Windows XP/NT sadly. :/

</comment></comment><comment><USER11> This is so cool. Would love to see some screenshots of the UI.

<comment><AUTHOR> Good idea! I'll take some today and add them to the github repo. Maybe that will be helpful. :)</comment></comment>


The Wilberforce Award - $1M for an alternative to consumption-fueled economy - <AUTHOR> http://dicksmithpopulation.com/wilberforce-award/

<comment><USER2> You have to be under 30 to win.</comment>


Landmark Math Proof Clears Hurdle in Top Erdős Conjecture - <AUTHOR> https://www.quantamagazine.org/landmark-math-proof-clears-hurdle-in-top-erdos-conjecture-20200803/ <comment><USER2> > the conjecture asks when an infinite list of whole numbers will be sure to > contain patterns of at least three evenly spaced numbers, such as 26, 29 and > 32.

A quick search didn't find the conjecture formulation, but in this form it looks suspiciously easy to prove wrong.</comment>


The Revenge of the IE Box Model? - <AUTHOR> http://www.jefftk.com/news/2012-02-18.html <comment><USER2> Notice that the author mentions CSS1 was published in 1996. IE3, the first version with CSS support was also released in 1996, meaning it was in development well before the W3C got together and decided on what they thought the standard should be.

It's not like they read the "standard", then deliberately misinterpreted it, as is the popular belief. Rather, they just picked a sane model and implemented it. If anything, it would have to have been the W3C who deliberately wrote their spec to leave IE out in the cold.

Naturally, I don't think either one was the case. It seems that the IE team made the correct choice (as it's very difficult to argue that the IE box model is not in fact a better way of thinking about boxes), and the standards team (who weren't actually in the business of building browsers) made a less good choice.

If only the Netscape team weren't so (possibly for good reason) angry with Microsoft, they might not have switched they way they sized boxes when they built Navigator 6, and we would have had a much less painful time writing web apps these last 10 years.

Shame. At this point, it's probably too late, as we're all thoroughly trained to deal with the idiosyncracies of the W3C box model to bother using the CSS hack mentioned in the article to get us back to a sane box model where a box with a width:100px actually renders as a 100 pixel wide box.

<comment><USER3> > Shame. At this point, it's probably too late, as we're all thoroughly > trained to deal with the idiosyncracies of the W3C box model to bother using > the CSS hack mentioned in the article to get us back to a sane box model > where a box with a width:100px actually renders as a 100 pixel wide box.

It's never too late to do the right thing. As an extreme example, everyone who's used to it now, will eventually die, and I'd venture to say the web will outlive them all. (although probably not in its current form, but that's kind of the point) CSS3's solution is a good one because it allows people that are used to it to keep using their old dodgy hacks, and it also allows people who prefer logical consistency to opt in to that. It's definitely not too late.

<comment><USER4> _everyone who's used to it now, will eventually die_

True, but there's a continuity of already-written code and learning, both from those alive now and from that already-written code, that keeps things that aren't sufficiently bothersome from getting changed.

</comment></comment></comment><comment><USER5> The real issue with which one "seems right" in some sort of intuitive sense is that it varies with what I'm doing. If I'm sizing something that is fundamentally a _container_ , then I care about the size of the whole box, including padding (and probably also border), for all the reasons that the pro-ie-box-model people have said.

On the other hand, if what I'm sizing is something that is fundamentally _content_ , then it's the W3C box model that seems more appropriate. "I want this paragraph to be 25em wide" probably means you actually want 25em worth of width actually dedicated to the letters themselves. "Scale this image to 320px width" makes a lot of sense if the image is natively some multiple of 320 pixels wide, and if the actual width the image itself gets is only 314 pixels due to border and padding, well, your image quality degrades.

The really-right solution is to be able to say "assume this box model for certain types of elements, and assume this other box model for other types of elements, and also let me override it on a per-element basis." Which the new CSS lets you do. Win!

</comment><comment><USER6> A quote from the article: "Logically, a box is measured from border to border. Take a physical box, any box. Put something in it that is distinctly smaller than the box. Ask anyone to measure the width of the box. He'll measure the distance between the sides of the box (the 'borders'). No one will think of measuring the content of the box."

I won't try to imagine the originator's reason for taking this view, but to me it undermines the reason for having a nonzero-width border between the inside of the box and the outside. I would have argued that the border of a CSS box is like the wall of a physical box, with an inside width and an outside width. Inside the border, we can adjust appearance with padding. Outside the border, we can adjust spacing between elements with margin. To a page designer, the "width" of the box should be the width of its interior.

Imagine a country that needs to establish the limits of its territory. There is a line on the maps between their territory and that of its neighbors. They decide to establish a border region, a finite-width "no man's land" (that only women can visit). Do they draw the "border" inside the established frontier, or outside it?

But these are all flawed arguments, and this is well-recognized in the container industry, who always specify a container's interior width, and an exterior width. To fill the box, one needs to know the interior width. To ship the box, one needs to know the exterior width. The difference is the wall thickness, and the corollary in CSS is the border width.

So the argument "Logically, a box is measured from border to border ..." is true, but would only be a useful description if the border had zero thickness.

But the argument that follows: "No one will think of measuring the content of the box", in which "content" is meant to contrast with "width", is obviously wrong -- to many, content size is exactly what measuring width means.

And as others have said here, it's far too late to be having this conversation. Standards must precede their application, not follow them.

</comment><comment><USER7> Yeah, I agree IE box model makes more sense. Padding is basically useless if it adds to width, most of the time you end up creating inner divs with margins to simulate padding. Didn't know you can toggle the behavior in Moz/WebKit, good to know!

<comment><USER8> supported in IE8+ too

</comment></comment><comment><USER9> IE's box model always made more sense than the Standard. Why would I ever want 100% width plus an additional 1px of border adding scroll bars to my page?!

</comment><comment><USER10> With 4 layers to the onion wouldn't it be easier to have 4 width measures? Then you could simply decide which width is most relevant to your application (outer margin, border, padding, or content).

There's different use cases, without listing them all just think how many times you've had to calculate the one you don't have access to

</comment><comment><USER11> Both models have an advantage. The padding outside the width makes it easy to make a border that doesn't touch the box, while the padding inside the box makes... well... padding a box easier.

Personally, I find the padding inside the box more valuable so when I found the `* { box-sizing: border-box }` post I decided to use it, and it's also what I expected it to behave like when starting web design.

</comment><comment><USER12> "[1] Technically the new rendering engine, Gecko, predates the Phoenix Firebird Firefox project. When Phoenix 0.1 was released in September 2002, Netscape had been using Gecko since November 2000's version 6. By that time IE was over 85% of the web, though."

Why do people always forget Mozilla (the browser, not the organization itself) ?

</comment><comment><USER13> Lets take a practical example.

Look at Uhaul 17' truck [http://www.uhaul.com/reservations/images/Equipment/Trucks/17...](http://www.uhaul.com/reservations/images/Equipment/Trucks/17Dimensions.png). What do you care most? Inside or outside dimension?

<comment><USER14> When I'm lining up trucks to fit in the lot?

The outside dimension.

</comment></comment><comment><USER15> IE did it right.

This problem gets exposed any time you try to do something like:

      width:100%; border:1px

Cause you can't see the border. It's really stupid, the spec got it wrong.

</comment><comment><USER16> >Adopting IE's approach here then might have told Microsoft "you can do whatever you want with IE and we'll adjust the standards to make it legal".

No, making an exception for this one case won't send that message. No one is telling them to change the standards for the rest of the quirks, many of which are arbitrary or just convention.

</comment><comment><USER17> IE's box model made more sense, but everyone else coded to the standard.

And yet, even IE9 has major CSS bugs.

For example, say you want rounded corners. Cool, it does border-radius!

For example, say you want gradients. Cool, it does ... well, sorta. ms-linear- gradient works fine on its own. border radius works on its own. But combine them together, and you're in for a world of hurt:

The box shadow, and border radius, show up fine. But the radius doesn't clip off the corners from the gradient - so you get a perfectly weird looking element.

I give them a B for effort, an F for actual usage.</comment>


What Nutmeg Can Tell Us About Nafta - <AUTHOR> http://www.nytimes.com/2016/12/30/opinion/sunday/clove-trees-the-color-of-ash.html <comment><USER2> This article seems like it's somewhat lacking focus. If anything, it's a brief foray into history of monopolistic wars that are the anti-thesis of trade agreements. What, indeed can Nutmeg tell us about, NAFTA though? Not much.

</comment><comment><USER3> I am from the state (Kerala) which is called the "Spice Garden of India". Little did I know about the history of Nutmeg that grows in abundance at my home in my state.

</comment><comment><USER4> This article tells us little about Nafta.

Also, it's written 'NAFTA'.

<comment><USER5> > Also, it's written 'NAFTA'.

The NYT style guide dictates that acronyms of five letters or more be capitalized as proper nouns (ie, first character capitalized, with the rest lowercase).

<comment><USER6> Seems very confusing and muddling to me, despite the appeal to "consistency". Sounds like 20th century style guide, not aligned with the need for specificity in the Internet age.

<comment><USER5> > Sounds like 20th century style guide, not aligned with the need for > specificity in the Internet age.

Ironic that you say that! The AP styleguide (which is not the NYT style guide, though they are similar) now recommends that "Internet" be lowercased (ie, no longer treated as a proper noun).</comment></comment></comment></comment>


Why headlines that aren't questions don't need to start with "Why" - <AUTHOR> It's unnecessary. An article is expected to explain "why".
How did this odd trend catch on?
Some recent examples from HN:
"Why Sleep Apnea Patients Rely on a CPAP Machine Hacker" -> "Sleep Apnea Patients Rely on a CPAP Machine Hacker"
"Why the world needs deep generalists, not specialists" -> "The world needs deep generalists, not specialists"
"Why I'm Giving $1.8B for College Financial Aid" -> "I'm Giving $1.8B for College Financial Aid"
"Why We Need Difficult Books" -> "We Need Difficult Books" "Why we rolled our own consensus algorithm" -> "We rolled our own consensus algorithm"
"Why Facebook's API starts with a for loop" -> "Facebook's API starts with a for loop"
"Why 536 was 'the worst year to be alive'" -> "536 was 'the worst year to be alive'"
"Why affordable housing is scarce in progressive cities" -> "Affordable housing is scarce in progressive cities"
"Why you shouldn't "work remotely"" -> "You shouldn't "work remotely""
"Why Google can't technically do customer support (2011)" -> "Google can't technically do customer support (2011)"
"Why New York Is Resorting to Paying Amazon for What Google Will Do for Free" -> "New York Is Resorting to Paying Amazon for What Google Will Do for Free"
"Why You Should Use a Text Area for Address Form Fields" -> "You Should Use a Text Area for Address Form Fields"
"Why Standing Desks Are Overrated" -> "Standing Desks Are Overrated"
"Why I Left My Big Fancy Tech Job and Wrote a Book" -> "I Left My Big Fancy Tech Job and Wrote a Book" <comment><USER2> This feels like a trend I can blame buzzfeed for.

Even if it isn't, I'm choosing to anyway.</comment>