How I spent two weeks hunting a memory leak in Ruby (2015) - <AUTHOR> http://www.be9.io/2015/09/21/memory-leak/ <comment><USER3> Interesting post, but I did want to chime in quickly and say that it's pretty absurd to only have 1GB for a web application in 2016, even for a small one. This is why I've been dismissive of using Heroku for my projects, even though I run a lean stack. They have higher RAM options, but they are incredibly expensive. For the cost of a 14GB Heroku Dyno, I can buy a dedicated server off ebay with 32GB ECC every single month. I don't use all of that RAM, but it's nice to have some extra RAM to toss around when you need to in a pinch.

I get that it means you have to "run a server", and _insert arguments for expensive cloud providers vs DIY servers here_ but I don't think it's any less crazy than being forced to chase a GC white whale for two weeks on a tiny memory leak to avoid a huge rate hike on your hosting bill.

On an aside, I've only had one problematic memory leak with ruby ever (the infamously leaky RMagick), I threw this in every time I used the lib and it solved it for me:

      GC.start(full_mark: true, immediate_sweep: true)

Bit of a performance hit, but not enough to cause a problem for my use case.

<comment><USER4> > it's pretty absurd to only have 1GB for a web application in 2016

While I don't for a minute doubt what you say, as an old guy, the absurdity of 1GB being insufficient for serving up web pages hits me pretty hard.

<comment><USER3> Our main proxies (that do the majority of the heavy lifting over here) run nginx and use, what, 50MB tops? If even? I throw them on $5/mo VPS instances and they just blast out IO without any problems.

But nginx was carefully written in C over the course of many years, with a general goal of hauling IO and static data around from other systems. Which is all you really need for web pages, in the end. Solved one thing and holy crap they did it well.

It's the business logic side of the web development world where things get nasty. The tradeoff is you get a giant chest of libraries for solving any stupid problem you want. I didn't have to for example figure out how to do IDN encoding for domain validation. You throw enough of that crap together in a single process and things start to get nasty. But how long would I have spent carefully writing all of it in C? Let's just say I'd be out looking for another job.

Worth the tradeoff, IMHO, but I think with projects like Crystal ([https://crystal-lang.org](https://crystal-lang.org)) in the future we will discover we can get pretty close to both in the end. That combined with Moore's law and I think we'll be good to go.

Assuming, of course, that cloud providers don't artificially restrict available RAM through economic constraint (know the true costs - <USER2> better!).

<comment><USER4> I realised (thanks to the other poster) that my comment may have seemed dismissive, which was not my intent. Just marvelling at it myself :-) But I do wonder about this kind of stuff. I have been wrestling with some legacy systems chewing up gobs and gobs of memory lately. Then wrestling with my own code that was chewing up gobs and gobs of memory :-)

It's pretty easy to make mistakes because we don't really get punished, as you say. But I wonder if we might be doing ourselves a disservice by not restricting ourselves. Generally speaking, better code is better and while it often _feels_ slower to write, my experience has been that it doesn't always work out that way.

I might try working in a memory restricted container to see if it improves my code...

</comment></comment></comment></comment><comment><USER5> About the patch at [https://github.com/vmg/redcarpet/pull/516/files](https://github.com/vmg/redcarpet/pull/516/files), couldn't the update simply have been the following?

        -	return Data_Wrap_Struct(klass, rb_redcarpet_rbase_mark, NULL, rndr);
        +	return Data_Wrap_Struct(klass, rb_redcarpet_rbase_mark, xfree, rndr);

<comment><USER6> The shortest code you can get away with is not always the best. If the parameter is a structure (which I think it is here, behind the void pointer), it's a good idea to give it its own free function in case later you add additional struct members that will need freeing/cleanup in the same place. (Bonus points if it appears in roughly the same place this thing is malloc'd)

I would have been even more explicit and added a cast to the expected structure pointer type. Entirely useless, except to the human reader.

<comment><USER7> That sounds like premature generalisation to me.

I'd say use a separate free function _when it needs one_ , which is not the case yet (and might never be.)

<comment><USER6> Call it an overreaction to a past project in which a predecessor scattered free calls over several uses, then I had to add fields.

I don't mind adding 1 useless function per complex type if it saves me those headaches even a small minority of the time, or for the next maintainer. Opinions may differ, but that's me.

</comment></comment></comment></comment><comment><USER8> shameless plug: I've made malloc hooking preload library for hunting such bugs, that automatizes most of the described work - [https://github.com/samsk/log-malloc2](https://github.com/samsk/log-malloc2)

<comment><USER9> This looks really, really cool. You should download the bugged version of this gem and make a screencast showing how your tool makes this process easier.

</comment></comment><comment><USER10> What a timely post.

I had a similar issue that I was tracing last week that did end up being my Ruby code...and it turns out I was modifying a constant like in the example.

What a fun read! I've been learning more about Ruby's GC since 2.1 and this got me looking even deeper -- definitely picked up a couple of new tricks/tools from this. Thank you be9!

<comment><USER11> Freeze your constants!

<comment><USER10> I used to do this with strings (now you don't have to), but this is great advice!

<comment><USER11> Yeah, it's one of those rare Ruby gotchas. If you assign some kind of a data structure to a constant, you'll get a warning if you try to reassign the constant, but the data structure itself will silently mutable. To disallow this behavior, simply freeze the data structure. If you have nested data structures, you must freeze deeply.

      ENUM = [0, 1, 2].freeze

      LOOKUP = {
        :A => [1, 2, 3].freeze,
        :B => [4, 5, 6].freeze
      }.freeze

      # or:

      LOOKUP = {
        :A => [1, 2, 3],
        :B => [4, 5, 6]
      }.tap { |s| s.values.each(&:freeze) }.freeze

</comment></comment></comment></comment><comment><USER12> There is no reason for using native extensions in scripting languages like Ruby when program load is pretty low (especially when they are used for performance reason, not binding).

Great insight, though. Author described this experience as it was a great venture... in hindsight I suppose :-)

<comment><USER13> Your criticism really applies to the gem authors, which to me, does not make sense. If I, as a library developer, want my library to be used on the critical path in heavy-load situations, and the library is for a language like Ruby, I'm going to figure out which part of my library are the performance critical parts, and implement them in a native language.

<comment><USER14> You would think that you (in this case, "you" == "average library author in the ruby ecosystem") would, but you won't. I'm fairly confident the problem is similar, but likely worse in the javascript ecosystem.

Necessity is the mother of invention, and when you're the first person to notice a memory leak, you're on your own.

Both ecosystems tout the low barrier to entry as a great benefit, but memory leaks, mostly benign inefficiencies, and poor algorithmic efficiency becomes at least par for the course if not a crippling liability when reality hits your application like a freight train (or say a 2 order-of-magnitude spike).

</comment></comment></comment><comment><USER15> I had similar experience of finding memory leak in Python. It's a 64 G server, but we load everything from database to memory. I didn't success.

This experience made me question myself: is it wrong to create a lot of objects in memory and let them reference each other, and hope Garbage Collector to magically work?

If instead of object, we just put data into memory more organized as a (column) database, then we don't need GC anymore.

OOP made programming easier, by modeling real world as object. At the same time, it made memory management harder, since it's not nature to the hardware's memory model: a linear array of memory cells.

I wish in future, there is a place for this paradigm of in memory database model of programming.

<comment><USER16> It's called "data-oriented design". Look into the gaming industry, where there's a lot of anti-OOP sentiment. They're primarily coming at it from the standpoint of optimizing memory access for cache lines, but there are side benefits that come from it around reducing memory allocation and usage, and avoiding the cost of abstractions that come with hidden inefficiencies.

</comment></comment><comment><USER17> Is there a simple way to notice your own subtle mistakes, when writing in a foreign language? The missing articles in texts from Slavic people bug me more than they should and I wonder what kind of (stylistic) errors I myself make.

The English grammar is mostly just simplified German, but there are some constructs, which I know of, but do not naturally use (e.g. "your _every_ …", "to name _but_ two") - they somehow seem twisted to me. Another one are compound words, I have given up on distinguishing between spaces, hypens and actual compounds and just use spaces almost everywhere.

</comment><comment><USER18> These are always interesting bug hunts.

I tried to add ASan to the travis config for this project but I couldn't quite figure out how to change CFLAGS and/or CC. Never used ruby but interwebs hinted that the bundle config/install commands might accept "\--cc" and "\--with-cflags" commands. It's ignored when I tried it though [1].

[1] [https://travis-ci.org/androm3da/redcarpet/jobs/159711520](https://travis- ci.org/androm3da/redcarpet/jobs/159711520)

</comment><comment><USER19> That is fantastic dedication to finding the source of the memory leak. I'm not sure I'd have quite the devotion or expertise to track that one down.

</comment><comment><USER20> Wouldn't it be useful if the community tested EVERY gem to see if it leaked memory, perhaps each developer stepping up to test one, in an app that just exercised the gem in question?

To test all of the ones we use individually is prohibitive. But thousands of people could do test one.

</comment><comment><USER21> A little worrisome that the fix hasn't been merged. _(edit: it 's been fixed)_

<comment><USER22> Did you get all the way to the end of the blog post?

"Update on 2015/09/29\. Redcarpet fix has been released."

<comment><USER21> oh, thanks. Disregard my previous comment.

</comment></comment></comment><comment><USER23> Great read. What's more, I found out that my company's codebase was still using the broken 3.3.2 version (code fix in the works).

Love it when reading HN pays off immediately like that :)

</comment><comment><USER24> Amazing stuff. I took time to understand this article by going through each part in detail. Learnt quite some stuff. Thank you for this incredible story!

</comment><comment><USER25> While the described debug process deserves nothing but my applause, I wonder - could the issue have been more easily diagnosed?

Ideally much less tooling would have been needed.

</comment><comment><USER26> Does PHP have similar issues? I think it shouldn't.

<comment><USER27> PHP extensions in C are pretty rare since it's so hard to distribute them (outside a few exceptions that exist in common package management systems), so in that sense, I guess?

But no, a PHP extension in C can have a memory leak like any other C code. It's not at all uncommon actually, since there are fewer consequences; CGI mode PHP cleans up after every request, FastCGI mode PHP by default restarts workers after a number of requests because of the class of effects (unwanted state) that includes memory leaks.

</comment></comment><comment><USER28> One you identified the leaked address I think a reverse debugger could have helped track the alloc down.</comment>


Gravity: An embeddable programming language without any external dependencies - <AUTHOR> http://gravity-lang.org/ <comment><USER2> I had the pleasure of contributing to Gravity a few years ago. I added builtin filter, map, reduce, reverse, and sort functions and tests to the array types. Marco was very receptive to my additions, and it felt very welcoming to newcomers. I learned a lot about different internal workings of interpreters making those contributions.

</comment><comment><USER3> I imagine this would be an alternative where embedded Lua gets used. A comparison with Lua would have been great to see. The 200K runtime, for example, indicates that this might weigh in somewhat heavier than Lua.

<comment><USER4> Agreed, Lua seems to be the de-facto standard here, as it's so lightweight to embed. But the language has some pretty janky rough edges (e.g. only having Tables and no other data structures) so a modern alternative would be nice.

<comment><USER5> Once you get to know Lua tables and then metatables, you will question the existence of different data structures :)

<comment><USER6> Yes, Lua tables are incredibly versatile tool. I'm not a fan of global-scope- by-default and of verbosity, but other than that it's a fantastic language.

<comment><USER7> I think global by default is one of the reason Lua shines in the embedded world. Not many other arenas where that is good practice.

<comment><USER8> Global scope by default isn't good practice in embedded. Global scope isn't synonymous with static allocation, but unfortunately that goes over the head of many people. Best practice in embedded is to have locally scoped statically allocated data passed around by reference.

<comment><USER7> It really depends on the application. For smaller, more targetted embedded applications, global scope is fine. If every function needs to have access to a common set of variables, there's really no need to be passing references around.

<comment><USER8> Even if the application is tiny, that's no excuse. It makes it even easier to chuck everything into a single struct type. It's not like there's any performance gain to having globally scoped stuff on a modern compiler compared to passing references.

One reason using global variables is bad practice is it makes testing harder. An unfortunately high percentage of embedded software doesn't have any sort of harness-based testing because it's written with globals spammed everywhere, which prevents you from using any kind of principled testing strategy where you mock out all the hardware dependencies. It's especially bad if there's globally defined MMIO stuff like "#DEFINE CCR1A _(uint64_t_ )0x74FEA10". Good luck testing that!

<comment><USER6> Smaller embedded targets don't have modern C++ compilers. Also many engineers want to solve domain problems instead of dealing with C++ related problems.

In domain of C, passing by reference means passing pointer. If you chuck everything into single struct and pass by pointer, it has same problems as global scope.

Not that I'm advocating for global variables. Even tiny projects tend to grow with time, and localizing scope across the code base is not fun at all. In context of Lua, I've just trained myself to prefix variables with 'local' and I don't give it much thought.

<comment><USER8> > If you chuck everything into single struct and pass by pointer, it has same > problems as global scope.

Not true. In fact, this refactor is one of the best things you can do to improve an old shitty embedded C codebase. Among other benefits, it allows you to have multiple instances (an arbitrary and easily-adjusted number, in fact) of a system sharing the same memory, reduces the complexity of linker-related BS, and simplifies testing. It's _vastly_ better than relying on horrendous C cross-module scoping rules for sharing.

<comment><USER6> To me the main problem of global is that any module can mutate and affect any other part. All encapsulation and modularisation is then leaky and you are always on your toes about implementation details in some other part of code. Your approach does not attempt to solve this downside of globals.

I agree that passing in structs is vastly better than communicating over globals. On the other hand, taking existing code base and implementing this state-passing is quite large undertaking that affects all function declarations/implementations. It might be beneficial, but there are often better investments of your time.

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER9> Slightly off-topic, but that's one beautifully done website.

<comment><USER10> The full-page pointless obstruction you have to scroll past to see anything useful is the opposite of beautiful to me.

<comment><USER11> To be completely honest with you, I didn't even realize I could scroll. That full-page landing page did have what I wanted, a single-sentence summary and a "get started" button to take me to the documentation, which loaded impressively quickly. I just found out that it was because it was on the same page.

</comment></comment></comment><comment><USER12> See also from 2018: [https://news.ycombinator.com/item?id=17373545](https://news.ycombinator.com/item?id=17373545)

</comment><comment><USER13> > Gravity has been developed from scratch for the Creo project in order to > offer an easy way to write portable code for the iOS and Android platforms.

... so, now we CAN have scriptable / interpreted code in iOS?

<comment><USER14> You always could, any of the thousands of React Native, Flutter or NativeScript apps are running a VM. What you can't do is run code that the _user_ provides, or use it to sidestep app store updates (with plenty of exceptions for both rules).

<comment><USER15> You also can't mprotect PROT_EXEC, which means no JIT.

</comment></comment></comment><comment> a-nikolaev The documentation says:

> Closures are self-contained blocks of functionality that can be passed > around and used in your code. Closures can capture and store references to > any constants and variables from the context in which they are defined. > Closures can be nested and can be anonymous (without a name)

I think, the term "closure" gains some unnecessary semantic meaning of a block of code / anonymous function. I might be wrong, but It is better to think about it as simply a technique for implementing lexical scoping.

In other words, "lexical scoping" is a property of a language, while "closure" is only an implementation detail to make lexical scoping work. So the term closure does not have to leak in the description and semantics of the language itself. What is your opinion?

Edit: I just think that such proliferation of terminology confuses people, making them ask questions like: "What is the difference between 1) function, 2) anonymous function, 3) lambda function, 4) closure?" Instead, focusing on the idea of a function (possibly without a name) + lexical scoping clarifies everything immediately.

<comment><USER16> Closures aren't necessary for lexical scoping, and in practice they're only real use (in imperative languages, obviously quite different in e.g. Haskell) is to pass around blocks of code combined with captured variables. I think this definition is quite reasonable.

<comment> a-nikolaev But such a closure block is a function (it might be sugared like a ruby block or something like that, but it remains a function that captures its environment, thus implementing lexical scoping).

Well, okay, if a language does not normally have lexical scoping, but has a special "closure" feature to implement it (dunno, maybe C++ lambdas may be regarded as such a special closure construct?) That could be a justification for the term closure on its own. But if the language has lexical scoping by default (e.g. Gravity says that it has lexical scoping in its overview), then, I think, there is no need in a separate notion of "closure" in the language semantics.

<comment><USER17> Lexical scoping doesn't necessarily include lexically scoped function definitions.

Lambdas have been spreading lately, but plain old functions are still not lexically scoped in a lot of languages otherwise regarded as such, C/C++/Java/Pascal etc.

<comment> a-nikolaev What do you mean by lexically scoped functions? A nested function?

My point is that instead of reusing the term closure, one could use the term anonymous function. To rephrase my question: Do you have a specific example where a nested function (or an anonymous function) is not lexically scoped (In a language that is otherwise lexically scoped)?

<comment> a-nikolaev Actually, I know Ruby messes up its functions this way:

        def f()
           x = 1
           def g()
             return x
           end
           g()
         end
        puts f()

        undefined local variable or method `x' for main:Object
        (repl):6:in `g'

So, the nested function g() cannot capture the scope. I think they had some justification for this funny behavior, because def is not a function, but actually a method (I could be wrong). But in any case, there are examples of languages that don't allow nested functions to capture environment (while a proc or block would capture it).

</comment></comment></comment></comment></comment></comment><comment><USER18> >Features >multipass compiler

irony is that this used to be a negative thing back in pascal days.

<comment><USER19> I used to mock Pascal. I feel mildly ashamed of this now. Turns out Pascal was simply a bit ahead of its time language-wise.

</comment></comment><comment><USER20> After having looked at it for a minute, it looks like a very simplified version of Swift. Looks great! Might try out the language on my Raspberry Pi and see how fast the thing is :p

<comment><USER21> "It is a class-based concurrent scripting language with a modern Swift like syntax."

</comment></comment><comment><USER22> How do you import a library?

<comment><USER22> Apparently you can't. I guess they're serious when they say "no external dependencies." On the plus side there's no need for a package manager.

</comment></comment><comment><USER23> Has anybody tried compiling this for an embedded target (e.g. Cortex-M4, arm- none-eabi)?

</comment><comment><USER24> Did anyone use Gravity before and make a comment about the ecosystem / community growth perspective?

<comment><USER2> I never really used it, but I did make some contributions to the language. In looking through some of the ecosystem, it seemed like the library ecosystem left a bit to be desired, but the standard library does show promise. I think its extensibility (proven by my ability as a noob to make useful stdlib contributions) and use cases in embedded applications mitigates the lacking ecosystem somewhat.

</comment></comment><comment><USER25> Cool stuff! Thanks for sharing.

</comment><comment><USER26> I am disappointed to see yet another language suffer from the same old semicolon cancer.

<comment><USER24> Calling it cancer is a bit hard. But I agree that semicolons are likely unnecessary.

<comment><USER27> (It's a play on "colon cancer", eh?)

</comment></comment></comment><comment><USER28> The first thing I want to know about a new language is its performance. Then I can start thinking meaningfully about the tradeoffs of using it compared to other languages.

<comment><USER29> For an embeddable language performance probably isn't the most important property for typical use cases.

Ease of integration, interopability with the "host" language, the embedded size of the scripting engine or compiler etc..., all might be more important.

</comment></comment><comment><USER30> The fact that it sets null to uninitialized variables is mindblowing for lang created within the past 5 years. We all have learned that there "null" is there just to create bugs.

Please Consider creating a Gravity 2.0 with this fixed as a compile error, you will save a lot of debugging time to your users.

<comment><USER31> Pure FP isn't the only acceptable programming paradigm, please stop pretending that it is.

(For context: a language without null pretty much implies an Option type, which is mostly useless (or no better than null itself) without static typing. Once you're that far you're probably gonna want to get monads to make them workable and congratulations you're reinventing Haskell)

<comment><USER32> Since when does static typing imply monads...

<comment><USER33> Actually, it does. If you want a sound and complete static type system for a practical language, you end up with monads (or something very close to it, like streams or effects) or you cannot type your side effects (and then your type system is either incomplete or your language impractical).

<comment><USER34> What does it mean for a type system to be "complete"? Type soundness means that well-typed programs do not "get stuck", but it is not obvious at all (at least to me) what it means for a type system to be complete, formally speaking.

If you just mean that a program has certain properties (like performing IO) that are not expressed in its type, then I can see what you mean. But even in Haskell, some properties like "this function might fail to terminate or throw an exception" are not expressed in the type (although you can use monads for those things in languages where all functions terminate by default). There are no distinctions between a function that performs network accesses and one that does not, or functions that never return odd numbers, etc. There are infinitely many program properties that you can come up with which are not expressible in Haskell types.

Even dependently typed languages with very expressive type systems cannot capture all possible program properties due to logical incompleteness results.

<comment><USER33> A type system is complete when every error-free program is well-typed. This is, afaik, seldom the case.

I was using the term more informally. In fact it is the operational semantics of your hypothetical language that is not complete in the sense that you will have a hard time putting it into a form that allows you to do anything meaningful (e.g. proof absence of certain errors) with it.

Interestingly, you _can_ define such a language with side effects (think ML), by actually pulling the state monad into your meta-level (for instance by turning your reduction relation into an instance of a state monad).</comment></comment></comment></comment></comment></comment>


Hardware is Expensive, Programmers are Cheap - <AUTHOR> http://lastinfirstout.blogspot.com/2009/01/hardware-is-expensive-programmers-are_29.html

<comment><USER3> Isn't the real story of this "incompetence is expensive"? Seriously, that application design is worse than the worst implementation we've ever had in a failed outsourcing project. (I make Big Freaking Web Apps for Japanese universities at the day job.)

Up until recently universities pretty much had to overspend on hardware, though. (Virtualization/expandable-on-demand cloud computing may eventually get around to changing it, but the customers aren't ready for it and the programming costs to take advantage of it dwarf the benefits for our clients.)

Most of the important systems that need to scale have very, very bad usage patterns from a hardware buyer perspective: for example, take course registration. (Edit to clarify: They are probably not talking about course registration, because there is no way in heck that peak is only 50% more than the steady state.)

At a university with 10,000 students, about 360 days out of the year you can run the course registration on a laptop while it is being used to play World of Warcraft. Then there is course registration season, at which point your peak concurrency goes from 1 user per hour to generally a _multiple_ of your student population all signed in at once. (Because, no matter what you do, they will open multiple windows/connections/etc because "the site is so slow, come on duuuuude, why the heck is this POS always so slow?")

All the accesses are dynamic. Most of them have writes attached. You have to get caching right because if you overbook a class and tell 15 students that they're confirmed a seat in a room which sits 12 because your cache got stale for three minutes, your customer gets yelled at, and they will turn around and yell at you. The end users are also typically incompetent at using the system (typically 1/4 of them have never used it before) and they will perform an impromptu fuzz test on it.

(Oh the stories I can't tell, sadly.)

<comment><USER4> I used to work at a company providing enterprise systems for higher education. Fall registration was panic season; we were in fire-alarm mode for most of August and September.

Those were good times. :)

</comment></comment><comment><USER5> Sometimes Hardware is Expensive, and so is time though.

I think often times in a startup, especially one thats bootstrapped, its easy to get caught up in trying to optimize everything too early on to scale for a million users before you have your first thousand. Sometimes you have to optimize for your development time first and product development goals, and leave the optimization till you get a bit of runway.

<comment><USER6> > Sometimes you have to optimize for your development time first and product

> development goals, and leave the optimization till you get a bit of runway.

That's a typical way to postpone and fall into a trap. Not all optimizations take huge amounts of time. And certainly there are some architectural decisions you can take early on that can save you a lot of wasted time later. The most common is to prevent bottlenecks.

For example, it's not necessary to make a clustered DB from launch day but coding and managing the UI and middle-ware to be ready for a switch to clustered DB usually takes very little impact. If you don't do this and the site becomes successful, re-engineering your whole site to support a new DB is usually close to impossible or extremely expensive. The site ends up in the DB hardware feedback trap just like in the article.

Edit: format fix.

<comment><USER7> Architectural decisions also limit what you can do later. Generally, the more focussed/optimized a system the less flexible. To the extent you KNOW exactly what the system should do, it should be optimized from the beginning; but, as PG points out repeatedly, most startups change direction at least once after launch.

</comment></comment></comment><comment><USER8> In other words, figure it out for yourself in your particular situation. Do the back of the envelope calcs. The reason why articles like this and the opposing view at Coding Horror are posted is because authors want to draw attention to management with preconceived notions.

</comment><comment><USER9> Looking at things along the hardware/programmers are expensive/cheap axis (take your pick) is short sighted and highly subjective. Setting aside political considerations, the real thing decision makers should focus on are the organization's underlying time and efficiency constraints and unfortunately these things are most often overlooked, misunderstood or inaccurately represented.

</comment><comment><USER10> This sounds like an edge case to me. Now that it's no longer 1998, most of us don't spec out $500,000 boxes for our stuff anymore. A $5,000 box will get you a long way for just about anything you need to do.

I suspect that the author is looking at a problem that would require tons of hardware regardless of how well it was optimized. Evidence of this can be found in the fact that even after tons of optimization, his $1.5M setup is still running at 20% load steady state.

Our single <$5,000 box handles about 4M pageviews per day without moving the cpu above 5% steady state. That's the sort of baseline I'm used to from the Microsoft stack, so it causes me to question whether the author is really looking at a mainstream case.

<comment><USER11> > Our single <$5,000 box handles about 4M pageviews per day without moving the > cpu above 5% steady state. That's the sort of baseline I'm used to from the > Microsoft stack

You realize how meaningless a statement like this is, right? You just can't go around talking about "pageviews" as if they were some uniform measure of workload.

<comment><USER12> Yeah, I was about to mention that. 4M a day is about 50/sec; if it's nothing but static pages you could serve that on a Pentium 1 without breaking a sweat. I've been giving away P4s recently, so their value is effectively zero - <USER2> can probably do it at under 5% too.

The problem is obviously when you're _not_ serving static pages.

</comment></comment></comment><comment><USER13> This is not a problem of hardware vs software. It's a problem of vendor's money vs your own. Of course he doesn't care. Actually, the more you spend on hardware the cheaper the software seems.

<comment><USER3> _Of course he doesn't care._

My day job sells integrated solutions to Japanese universities. We would care very intensely how much the hardware costs in a circumstance like this. (Which we wouldn't be in, because we try not to deliver software which is an abomination against all that is good and holy in terms of database use, but still.)

The math is simple: the university typically has a budget of X million yen to Get This Done. It doesn't matter what the line items are on our invoice -- we can't charge them more than X million yen.

Given that constraint, what do you think we want to charge them? Software license fees for our solutions, where our margin is anywhere from... crikey, I can't tell you the numbers, but "high to higher". Software license fees for third party providers like a certain Enterprise Database, where the margin to the reseller (us) is from low to medium? Or the margin resellers get on hardware, which compared to our software is small enough we could use it to remove food from between our teeth?

<comment><USER13> I mean, once the contract is signed. I know how sucky it is to go into such a relationship with a vendor. Custom apps are always risky... and hardware costs are one of the risks.

In your situation, actually in most situations when the client can compare the costs for both software and hardware before, it's logical to optimize the software.

</comment></comment></comment><comment><USER14> there is a fundamental difference between software for sale and software as a service when it comes to this debate. the economic efficiency math is much different for licensed software because more than one company is using it.

<comment><USER15> "the economic efficiency math is much different for licensed software because more than one company is using it"

So is the risk.

Nothing drives customers crazier than a slow system because _someone else_ is having a busy day.

I spent my first 10 years decoupling applications from each other because independence was more important than economies of scale. Now we're swinging back the other way. Hopefully, we will have learned something this time.

</comment></comment><comment><USER16> Yeah, if you can make software run 100 times faster with lots of profiling and hand optimisations, that $300 laptop is basically a $30,000 laptop.

<comment><USER17> Of course you need to consider what those optimisations will do to maintenance and future development costs.

</comment></comment><comment><USER18> I think the problem is that the costs add up slowly. Your site starts getting slow? You just throw another dedicated server at it, increasing your cost 300-400 bucks a month.

</comment><comment><USER19> Hm, why not find something to do with the remaining cycles?

<comment><USER6> In a production database!?

<comment><USER19> I was assuming in a modern "cloudy" setup, processing power and storage is all kind of fluent. Or if it isn't yet, might be worthwhile to make it so? They could set it up all with virtual machines and stuff?</comment></comment></comment>


Cyprus may raise bank confiscation to 25% on uninsured deposits (&gt€100,000) - <AUTHOR> http://www.nytimes.com/2013/03/23/business/global/cyprus-bailout-vote.html?pagewanted=all

<comment><AUTHOR> _(Who do you report bugs to on HN? In titles, the character " >" is silently erased: you need to manually enter the HTML entity "&gt". This can't be intended behavior! (Apologies for being off-topic, I really don't know.))_</comment>


Algorithmic Interviewing – Optimizing to Hire the Wrong Developer - <AUTHOR> https://medium.com/p/9a3781d57c4b <comment><USER2> _" Being a great business web developer and being a great algorithm developer are not mutually exclusive, but they are also not interdependent. The type of developer that would perform well at this kind of challenge is potentially a recent graduate, fresh from an algorithms class or one that is more focused on micro-optimizations than the big picture."_

Oh, good grief. Being able to answer basic algorithmic questions in an interview isn't a high bar, folks. It's not as if a basic tree-traversal question is going to trip up everyone but Don Knuth.

The biggest danger you have as an interviewer is allowing a Stealth Incompetent on your team. These folks all have Github profiles, points on Stackoverflow, fancy blogs, webpages, etc. They might even have nice-looking, public-facing projects. You can even argue that these people Get Things Done (for some value of "things"). But they're still bad hires.

Stealth Incompetent programmers are great at using other people's code. They can install gems, download jQuery libraries, and stick them all together. They can make websites with Rails or Django or some other framework, and know enough to be able to piece together functional "products" from parts. All fine. But unlike actual engineers, Stealth Incompetents fall on their face the first time you ask them to do anything that requires taste or independent thought. They're the people who will introduce a horrible dependency on a bad gem, or architect a solution that takes quadratic time, when they could have trivially done better with a tiny bit of thought.

Algorithmic questions are designed to _catch_ incompetent people before they can infiltrate your team. The "relevancy" of the questions to the day-to-day work is immaterial.

<comment><USER3> I would argue that, say, a recent graduate who can absolutely nail algorithmic questions would be the _most_ likely to screw up on the big ticket items that matter (infrastructure, taste, independent thought) precisely because they have zero real world experience.

<comment><USER2> _" a recent graduate who can absolutely nail algorithmic questions would be the most likely to screw up on the big ticket items that matter"_

Why do people keep implying that only new grads understand algorithms as if it's some kind of fact? In my experience, new grads don't tend to know algorithms any better than they know anything else.

Good engineers keep learning about algorithms because the knowledge is timeless.

<comment><USER3> Ridiculous. Good engineers solve engineering problems. The huge number of successful engineers out there in the world solving problems without remembering their CS course notes tells you everything you need to know about how important much of the algorithmic subject matter is.

<comment><USER2> Uh huh. I can fix my car without knowing metallurgy and fluid dynamics, but you wouldn't want me building you an engine.

Just because people are getting by doing the technical equivalent of changing sparkplugs doesn't mean that I want to hire them to do something more difficult.

</comment></comment></comment></comment></comment><comment><USER3> This exact topic came up yesterday in the HN post "Bullshit Interviews": [https://news.ycombinator.com/item?id=6304911](https://news.ycombinator.com/item?id=6304911) ([https://coderwall.com/p/yn9g2a](https://coderwall.com/p/yn9g2a))

I repeat now what I said then: I think there's a very simple rule anyone can and should follow to not fuck up interviews: Don't ask people to do things in the interview you wouldn't expect them to do in the job. Examples:

(1) Do people in this job regularly write code on the whiteboard? Lack access to a search engine? No. Then don't do it in the interview.

(2) Do people in the job design spice racks for blind people? No. Then don't ask them to do this in the interview.

(3) Do people in the job regularly find themselves coding something on the level of fizz buzz? Yes? Go ahead and ask!

(4) Do people in the job regularly find themselves writing code to find loops in a graph? Yes? Go ahead and ask (though, let's be honest - this is _exceedingly_ rare in comparison to how often it is deployed in interviews)

Verify people have the skills they need to do the job you're hiring for. It really is as simple as that.

</comment><comment><USER4> I find it curious how so many computer engineers tend to have borderline religious beliefs when it comes to the topic of CS fundamentals.

This is an ineffective way to hire front-end devs, which is the scenario described in the article. In Javascript, there is limited room to implement fundamental algorithms, because many of them have already been implemented in native code, and they will pretty much always outperform a customized Javascript implementation for large n.

When doing front-end dev for a living, you will not be able to keep your fundamental algorithms chops by just doing your job. You have to go out of your way to keep them fresh. As a result, interview questions about fundamental algos for a position like front-end development will be biased in favor of those who are recently took CS algorithms classes, who do not necessarily have any more intrinsic potential as a front-end web dev, and who likely have less work experience.

As a consequence, it would seem that any employer that uses fundamental algorithms as a major factor in hiring front-end devs does not have a realistic conception of what the job actually entails.

And the same argument applies for any programming job where fundamental algos are not coded regularly.

</comment><comment><USER5> I have mixed feelings about algorithmic interview questions. There is the camp that says that you shouldn't memorize something you can look up, but I am of the opinion that having something memorized fundamentally changes the flow of thought.

If you just _know_ the lookup times of various datastructures, your thought process is much less jumbled than someone who must look them up while problem solving.

I'm not suggesting that it's necessary for someone be able to whiteboard an implementation of a red-black tree, but knowing what it is, and its advantages and disadvantages compared to an AVL tree or something is very useful.

As an aside, in a graduate CS class I took (about search engines), the prof asked "What is the lookup time of a balanced binary tree" to various students, two of whom gave wrong answers before she got to me. One said "n lg n" and the other said "n". It was evident that they were basically guessing.

My point is that if you actually understand what a binary tree is, the lookup time of it should be obvious. Same for most other basic datastructures, linked list, hash table, etc. I think asked basic, high-level questions about these are totally fair game.

<comment><USER6> [pedantic] Technically, look up on a balanced binary tree is still O(n) as it says nothing about the ordering. If it was a balanced binary _search_ tree then it would be O(log(n)) which I guess is the answer you gave. [/pedantic]

A part from that I totally agree with you on the importance of knowing by heart the implications of some technical choices.

<comment><USER7> (I upvoted you because you're pedantically correct, but being even more pedantic: if it's not a balanced binary _search_ tree, is "lookup" even a valid operation on it?)

<comment><USER8> Well, lookup is a valid operation on any dataset.

<comment><USER7> Not really: "lookup" is not part of the API (as taught in a typical data- structures class) for stacks, queues, and priority queues, and technically arrays/vectors let you lookup by integer but the O(N) lookup by object is only an extension that most languages provide as a convenience.

If we really want to be pedantic, we'd have to define what the binary tree is used for, because there's no convention for what "lookup" means in this case. A heap, for example, can be implemented as a balanced binary tree, but the only methods it provides are push, pop, and top.

</comment></comment></comment></comment></comment><comment><USER9> I'm in a school of thought that algo interviews are useful, but not for the knowledge of some particular algorithm.

I was usually asking people to write atoi() implementation without using high- level constructs on the whiteboard, piece of paper or in the browser window. Things I was looking for:

1\. If I'm interviewing for C# position and a person chooses VB.NET, that's probably a red flag.

2\. If they choose C, hey! show me your pointers arithmetic, strlen() at the beginning of the loop is a huge red flag.

3\. There're always giveaways about how person is familiar with the language (and mind you, I do not expect any tricky stuff to be used).

But the important part is not the code - the important part is to watch how the person attacks this token problem, how much time do they spend on it, where they're when I come back in the room. Then we'll talk about what the did in order to solve the problem and why did they do things they did. I won't be running this problem to check if the output is correct or to see if it compiles at all - we'll just talk it over. Will we talk about technology stack in use at the company, assess person's architectural skills? We sure will. But at the end of the day I'll expect this person to write some code at their job and I need to be sure that they know what they are doing.

Does Github account help? It does to a degree. I'll sure take a peek to see at the coding style or how the projects/code are structured, but honestly, I just don't have time to fully assess someone's capabilities by reading their existing code thoroughly. Plus, it doesn't give me any clues on the thought process when they were writing that code.

</comment><comment><USER10> Every now and then, these posts come up on HN people mention how bad the interview process at certain companies is and how it should be changed.

I also work for a company that does these types of interviews. While I wholeheartedly agree that asking these algorithms/data structures or other CS fundamental questions is not perfect, precisely because it may induce quite a bit of false negatives, the engineers who does end up successfully completing the process at my company are very competent ones. So from an anecdotal perspective, the process works fairly well in the fact that it does not produce too many false positives.

<comment><USER11> There's a broken culture of companies thinking they should only hire the smartest people in the world. Most companies don't need brilliant people, they need competent people who will work great together.

[Disclaimer of bitterness: I'm not one of the smartest people in the world. I get rejected after one to three interviews all the time. The author of the post is right: nobody ever looks at your online code and your resume only gets read once by HR to set up the initial pre-interview.]

Testing for concrete intro-to-CS level drivel may give you a company full of people who spend 30 hours a week doing topcoder or project euler, but... you don't need that (if you do need that, good for you, but most companies don't).

It also tricks the interviewee a bit -- "Oh, they're asking algorithms questions! Maybe this'll be a fun CS research job." [six months into the job] "Uh... all I've done is write SQL and fully library-driven backend website code. Boring."

</comment></comment><comment><USER12> I strongly believe that for many companies such problems with hiring process start much earlier - in the phase of defining the requirements (competencies). When they don't know _precisely_ what they need (and both "ninja" and "experienced front end dev" are far from being precise), they simply can't succeed in measuring it.

</comment><comment><USER13> If you're looking for people with the proverbial "smart and gets things done" properties, you need to give them something to do and see how smart their solution is.

I give a task, like "write some code to store data in X; here's the interface I'd like to use" or "write some code to find X given data set Y". The problem should have "layers" and follow up questions.

Some devs recognize the task and says "this looks like an application for $FOO" and apply some $FOO. Others do not know the name of the optimal technique, but are able to derive it on the spot from first principles.

Junior devs should at least be able to come up with a brute force solution that solves the problem, even if they can't take it to the next level. If the problem requires some deep "ah-ha" moment then it's not a great test; the whole point is to separate applicants into strata.

</comment><comment><USER14> "My experience gave me the feeling that this was more of an interview assembly line than someone really interested in my relevant skills"

That's exactly what it is...

Effective interviewing is hard, and like most things hard you won't be very good at it without some practice and study.

Asking an algorithm question is relatively easy.

</comment><comment><USER15> I was recently interviewed by Google, didn't get the gig, didn't do very well If I'm honest, however even though I'm a bit bitter, I'll admit that knowing algorithms and data structures is important. I was rusty before the interview so I hit the books and I was amazed how many things I've forgotten and I do believe knowing those thing makes you a (much) better developer. Whoever says otherwise, do yourself a favor and learn at least the fundamentals.

</comment><comment><USER16> Aside from the main point of this article, could anybody recommend a good algorithms/data structures book?

I want something that gives examples of _when_ you should use particular techniques/theory and the _benefits and trade-offs_ of doing so? That is, as opposed to learning the rote implementations and time complexities, I want to understand the practical application of knowledge...

I recently looked at some coding tests for Spotify [0] and found an interesting article talking about the solution for one of these. It was interesting to me since he just pointed out that something was a maximum cardinality problem on a bipartite graph and then applies the Hopcroft-Karp algorithm [1]. Now I don't care at all about being able to code any of those things from scratch, but I'd love it if there was a book which I could read to try and learn the names of different techniques and when/why they are applicable. I want a better map and I want it to be written in english.

Any ideas?

[0] [https://www.spotify.com/uk/jobs/tech/catvsdog/](https://www.spotify.com/uk/jobs/tech/catvsdog/)

[1] [http://alonso-vidales.blogspot.co.uk/2013/03/new-spotify- puz...](http://alonso-vidales.blogspot.co.uk/2013/03/new-spotify-puzzles- reversed-binary.html)

<comment><USER17> Try Skiena's "Algorithm Design Manual" [0]. Another nice book would be "Programming Pearls" [1].

[0] [http://www.amazon.com/Algorithm-Design-Manual-Steven- Skiena/...](http://www.amazon.com/Algorithm-Design-Manual-Steven- Skiena/dp/1848000693) [1] [http://www.amazon.com/Programming-Pearls-2nd- Edition-Bentley...](http://www.amazon.com/Programming-Pearls-2nd-Edition- Bentley/dp/0201657880)

</comment></comment><comment><USER18> I think my answer to the "write a sort" thing would be "are you kidding? I'd use the one in the language's standard library, because I'm not an idiot. Or if I had no choice, and was using a language and environment so primitive it had no libraries at all, I'd write bubble sort because it's easy. Or if it actually mattered I'd go read up on it and learn how."

<comment><USER7> Oftentimes that's the _first_ answer that the interviewer is looking for (without the attitude). But then they want to see you code a MergeSort or QuickSort up from scratch, and explain the difference between them.

Why? Because what happens when your data set exceeds available RAM? Then you can't use the stock libraries, since they're all designed for in-memory datasets. You have to rely on external sort, which is a cousin of merge sort but has fairly different properties because of the vast difference in performance between RAM and disk. Or what if you have multiple computers available to you? Do you implement a parallel external sort over the network as a library (or service), or do you rely on domain knowledge of the keys to do some sort bucketing to each computer, like a radix sort? What are the trade-offs between each one?

Software engineering is somewhat peculiar in that 95% of the time is spent gluing pre-made components together, and yet the majority of the money is made off the 5% that requires actual creativity and knowledge. Why? Because gluing components together is a commodity. You can do it, but so can every other college grad who knows how to Google StackOverflow answers on the web. The folks who make $300K+ a year at Google/Facebook or get acquihired for a mil or two a piece or found a startup that strikes it big are the ones who know how all the "standard" stuff works on the inside so they can take it apart and piece it together in new and innovative ways.

</comment></comment><comment><USER19> Did we all read the same article? He complains about "algorithmic interviews", the comments here are about AVL trees and sorting algorithms, but what he was actually asked to write was "a string parsing implementation".

Well, OK, I suppose it's possible they tried to get him to write yacc, but it sounds much more like they asked him to do some trivial loops and tree traversal, at the very most some basic recursive descent parsing.

This ain't rocket science, it's not complex O(n) analysis, it's only slightly above the fizzbuzz level. If you're writing code at all then I think it's reasonable expect you to understand recursion and loops, even if much of the job is just going to be hooking up jquery plugins.

</comment><comment><USER20> I don't agree 100% with the author on not asking them algorithmic questions. Its a process on how they go about solving it rather than just the outcome. Maybe for the web developer position, you might not need such type of questions, but for a full stack developer the answer is unequivocally yes.

I made a mistake by hiring someone who knew django [who originally wrote plugins in WP] and the coding was just plain horrible. One such example.

`for foo in queryset: foo.prop = 'i m bar' foo.save() ` Simple things like this, where a good engineer would know because of O(n) nature of the code, the so called programmer didn't.

And then there are guys who fork every possible repo, have blogs about how- tos, etc but really can't code for nuts.

<comment><USER21> Coding noob here. Can someone explain the problem with the

`for foo in queryset`

implementation? Is it that for loops are inherently inefficient? What would be a better alternative?

<comment><USER22> I don't see a problem with an O(n) algorithm to set some property on n objects.

<comment><USER23> I think the issue is the call to save() (I assume this is meant to be outside the comma).

Basically, it'll hit the database after each query.

<comment><USER21> I originally thought it might be that too, but it seems to me like you'll have to hit the database n times for n queries no matter what. Unless Django has some builtin batch save functionality.

<comment><USER23> Hmm, there's bulk_create(), introduced in Django 1.4, to create multiple new objects in one DB call.

However, I'm not aware of anything that would do the same to update properties.

</comment></comment></comment></comment></comment></comment><comment><USER24> The argument about whether to ask algorithmic questions reminds of the "can an airplane on a treadmill take off?" question. There are two sides who arr sure that they're right and the others are wrong, but they're just answering subtly different questions.

In this case, the people against always seem to come back to questioning why you would ask something that's just memorization, while the pro-algorithms side is pushing for questions that candidates wont know the answer to, so they can see how they think.

I am in the second camp, if you cant tell.

</comment><comment><USER25> I personally don't look for someone to answer algorithmic questions correctly. If they can just show the ability to solve a problem with a bit of complexity, that's all that counts. It also just reveals a lot about how that person thinks and solves problems which is absolutely an important factor of a potential job-hire. However, I know some interviewers just keep score of your answers..I don't necessarily agree with that.

</comment><comment><USER26> Do you hire a mechanic who knows how the car works in intimate detail over a mechanic which knows how to fix cars? Depends. But usually if they know how the car works chances are they are also great at fixing them.

In tech it is the same, with the additional wrinkle that you want to hire aptitude over experience.

</comment><comment><USER27> But algorithm coding is just about personal programming ability. What if the company is using a non-opensource lib? I don't know how difficult the string parsing problem is. But interviewer may just unsatisfied about your slow implementation.

</comment><comment><USER28> I think algorithmic interviews are fine if the interviewee has mentioned the algorithms on the C.V.. In general, I only talk about topics the interviewee has mentioned on the C.V. or are super important for the org/project/position.

</comment><comment><USER29> Interviewing process is constant optimization just like every other workflow process in a company.

Disclaimer: I run interviewstreet.com which helps companies screen & hire programmers using coding challenges. We have had a substantially HUGE number of developers screened (might not be able to reveal the exact number) through the process and using the data points, here's what we found (btw, this is also constantly optimized)

a. Asking a 5-year experienced engineer to solve a graph theory challenge or a complex tree problem is a pretty useless indicator. The goal of such an interview challenge should be to test the problem solving skill of a candidate which is essential across any programmer role.

Can this person actually take an array of objects, perform an operation to get the result such that it works for any size of the array without throwing an exception? The data structures used in the question should be simple enough to start working on the challenge. You will be surprised how many errors, corner cases come up which are often missed. And as you gradually increase constraints, you can check their thought process of how their algorithm changes.

b. Make the problems interesting - put in actual effort to make the problems interesting, sometimes relevant to the problem you are actually solving. I often send this link to our customers(www.itasoftware.com/careers/puzzle_archive.html) and also help them design problems like this. That's by far the most interesting publicly available challenges. The other one is Quora, quora.com/challenges which is slightly harder though but very interesting (and has proved very effective!)

c. Calibration: Surely github/bitbucket/SO profiles are important and can serve as a data point. However, it's probably going to be very hard to calibrate. A web server coded in Python vs a new MVC framework written in PHP - who is a better candidate? Who has actually thought through the design of the problem better? It's tough to evaluate. The programming challenge interviews act as a data point (one of the interview rounds) to check these skills in a contained problem/environment helping the company calibrate the performance against the rest of them.

The problems should involve a combination of the ability to write good code, focus on problem solving skills and not on remembering an algorithm from the CLRS text book, intelligent thinking (not to be confused with weird math/geometry problems unless your company is working on that domain) and _importantly_ the ability to catch hints and solve it in a better way. This is a data point, an essential one but there are more to technical interviews.

If used in the right way, it can prove to be hugely effective for your process not only in streamlining but also as a way of generating interest from potentially interested programmers.

<comment><USER3> I am actually pretty decently ranked on Interviewstreet because I do have a personal interest in algorithmic stuff (I was top in Australia for a little while :)). I used to do your codesprints just for fun.

My feedback to you would be that Interviewstreet is heavily heavily geared towards academic algorithmic questions which the average developer would not have a chance in hell of solving. For example, in order to solve some of your old Hacker Rank questions within the time and memory constraints you pretty much have to know the studied and documented solution you would only ever be exposed to in doing a masters degree in CS. Heck, your problem descriptions were often thin veneers around the exact academic statement of the problem.

Despite the fact that I'm someone who actually does excel at these types of problems, I think they are absolutely the wrong way to hire people and whenever I've seen companies actually use Interviewstreet it makes me cringe - and yes I know companies set their own questions, but you lead by example and so most questions are abysmal.</comment></comment>


The Real Threat to Google - <AUTHOR> http://www.businessweek.com/technology/content/apr2008/tc20080427_580014.htm?chan=top+news_top+news+index_businessweek+exclusives <comment><USER2> Simple: intersperse ads with results and make it obvious which are the ads.</comment>


Awesome-indie – Resources to help independent developers make money - <AUTHOR> https://github.com/mezod/awesome-indie <comment><AUTHOR> I was quite surprised that there wasn't a list yet, so I started it :D

Feel free to contribute with quality content!</comment>


Will Android game development ever catch up to iOS?  - <AUTHOR> http://www.fiercedeveloper.com/story/will-android-game-development-ever-catch-ios/2011-04-18 <comment><USER2> On an iPhone (or any other current iOS device) you can know that you have nearly all, if not all of the processing power available (OS does notifications, wifi, etc).

On an Android you have no idea how fast the processor is, no idea how much there is going on in the background and so on. You will have to aim lower for the game to be usable on an average Android, while an average amount of other applications are running.

Possibly not on topic, but true and relevant.

<comment><USER3> You have the same issue in the PC gaming world and it's thriving.

<comment><USER2> Complex settings of quality. And questions like Will It Run Crysis.</comment></comment></comment>


Quantum Number Generator - <AUTHOR> http://photonics.anu.edu.au/qoptics/Research/qrng.php

<comment><AUTHOR> Other relevant links:

Live hexadecimal stream <http://126.96.36.199/RainHex.php> (They also have a binary and "color" streams, as well as static info.)

A (small) article in phys.org <http://phys.org/news/2012-04-fastest-random- silence.html>

A prior HN entry (<http://news.ycombinator.com/item?id=3470490>) about a "quantum number generator", from a different lab: <http://qrbg.irb.hr/></comment>


Why you should always read user comments under posts - <AUTHOR> http://technicallyeasy.net/2012/03/why-you-should-always-read-user-comments-under-posts/

<comment><USER2> There's nothing like wading through sewage to find the occasional lost gem. Personally, I'm increasingly finding that the ROI in reading comments is close to zero.</comment>


Olympics Committee Says Non-Sponsors Are Banned from Tweeting About the Olympics - <AUTHOR> http://gizmodo.com/olympics-committee-says-non-sponsors-are-banned-from-tw-1784344194 <comment><USER2> The Olympics have always been protective of their brand. I was doing some research a few weeks ago because my start-up ([https://doarama.com](https://doarama.com)) is creating some 3d maps of events at the Rio games and wanted to make sure I wasn't going to get on the wrong side of the games.

My research resulted in this discovery that the Olympic rings are no longer under trademark protection (according to Wikipedia) [https://en.wikipedia.org/wiki/File:Olympic_Rings.svg](https://en.wikipedia.org/wiki/File:Olympic_Rings.svg) I don't see how they can ban anybody from tweeting about the Olympics as it is news.

It would be nice for the Olympic committee not to just throw out these empty threats.

<comment><USER3> Sorry buddy, the IOC like Disney get what they want (or pay for):

[http://www.inta.org/TrademarkBasics/FactSheets/Pages/Protect...](http://www.inta.org/TrademarkBasics/FactSheets/Pages/ProtectionofOlympicTrademarks.aspx)

</comment></comment><comment><USER4> The Olympics aren't hugely relevant anymore. We'd all be far better off just ignoring the games and focusing on the corruption that establishes them in the first place.

</comment><comment><USER5> What could possibly go wrong with this :)

It will not only work, is non-enforcable, it is also totally against the spirit of the olympic games.

</comment><comment><USER6> Double-plus good!</comment>


How The Martha Graham Google Logo is Animated - <AUTHOR> http://www.acumenholdings.com/blog/how-the-martha-grahm-google-logo-is-animated-short-explanation/

<comment><USER2> I can't see any doodle on Google homepage, don't know why it's not available to my country. Here's a higher resolution video of the doodle by the artist Ryan Woodward himself:

<http://www.youtube.com/watch?v=IsbI1GE4DGA>

It appears that animating dance figures is his speciality, check also this:

<http://www.youtube.com/watch?v=OBk3ynRbtsw>

<comment><USER3> It has been taken down. Their doodles last only 24 hours.

For future reference, if you think that a Google doodle is not available in your country, you can try www.google.com/ncr

<comment><USER4> Saved mirror anyone? I missed it.

<comment><USER5> They usually end up on <http://www.google.com/logos/> after a few days, time permitting. There's a backlog right now because there've been like 5 animated doodles in the past month.

</comment></comment></comment></comment><comment><USER6> I can't access the article but looking at the homepage, I'm surprised it's not just a giant spritesheet that changes the x/y offset on certain intervals.

Nevermind, it is a giant spritesheet, but it's optimised to allow for static elements to not have to be redrawn. <http://www.google.com/logos/2011/graham11-hp-sprite.png>

<comment><USER7> Yes, but isn't Google supposed to be able to come up with an elegant equation to represent those coordinates so it's not just this huge ugly pile of crap? I popped the code open on it this morning and yawned. You know a company's starting to get love handles when they have people who spend a week doing stuff like this that runs for a day, mildly annoys a bunch of people, and is immediately replaced.

<comment><USER8> You do understand that people are literally going to google to watch this animation, play pac-man and whatever else cool that happens on the frontpage? Made it to Hacker News frontpage and people are talking about Google's homepage. Job well done.

<comment><USER7> Lighten up. People go to google regardless. And people talk about it because it's the first thing everyone sees when they get to work in the morning. All I'm saying is it's not really much of an accomplishment. An animated gif would have been the same size file, for example -- no giant array necessary. I'm just saying it's ironic that a company that's supposed to exemplify elegant systems would end up writing single-use code that basically boils down to a large ANSI graphic you'd see on the front of a WWIV BBS.

Hugh Hefner's birthday is April 9th. Where was his banner?

</comment></comment></comment></comment><comment><USER9> To get an animated feel for how this works, add a user CSS to outline the border. I use the "User CSS" extension for Safari

<http://code.grid.in.th/>

Then add this CSS:

#hplogo div { border: 1px red solid; }

</comment><comment><USER10> I wonder why they went with many <div>'s, instead of a single <div> with JS controlling the background-position and clip?

<comment><USER11> With many small divs they only need to redraw the part that changes. The sprite can also be smaller because every frame doesn't have to be the full size of the logo.

</comment></comment><comment><USER12> Select the logo as it animates to see more clearly how it's a bunch of divs being spawned. (To select it, either use your mouse or tab out of the search box and hit Ctrl-A)

link: www.google.com

</comment><comment><USER13> The animation was cool the first time, then after that it just got annoying - like having a flash ad on the page. I changed my search to Bing for the first time ever because of this animation.

</comment><comment><USER14> Let me guess : by taking up 100% CPU? Irritated me the whole day.

</comment><comment><USER15> I use igoogle start page, and I can't see any animation, just a static Martha Graham Google Logo. Does anyone have a the correct google URL in the States? I've seen the video of it, but it annoying me that I can't see the actual animation.

*edit: I found it <http://www.google.ca/>

<comment><USER11> if you don't want to log out, just open up chrome in porn-mode (incognito) and go to google.com

or just use any other browser your'e not logged in to. or just log out.

<comment><USER15> I guess since google.com/ig stores a cookies on my browser, all of them actually, I couldn't get the normal google search page, even with logging out. Thanks for the private mode tip.

</comment></comment></comment><comment><USER16> The logo didn't work in Opera 11.10 :/ I had to fire up IE 9 to take a peek

</comment><comment><USER7> Ain't really brilliant. And they used the same 'doodle' method before. What would be nice is if they used the HTML5/canvas methods they've been hyping so hard. Unfortunately, animating this with that would take at least a QC Xeon.

What would be really, really awesome would be if they did it with Flash. Then it would work on droid, and iOS users would all go 'wtf?'

<comment><USER17> Yeah I wish they would use canvas for browsers that support it, and fall back to the million-div method

<comment><USER5> Bunsen was done with canvas:

<http://www.google.com/logos/bunsen.html>

It probably wouldn't have worked here, at least not without the gorgeous artwork that makes the doodle famous. It was already a huge PITA to get all the control points into Bunsen, and that was a mostly static animation with pretty much simple polygons in motion. Martha Graham has fluid motion and semi-realistic human figures; this is far too complex to draw with just code.</comment></comment></comment>


Panama Papers Reveal Clinton's Kremlin Connection - <AUTHOR> http://observer.com/2016/04/panama-papers-reveal-clintons-kremlin-connection/ <comment><USER2> Is this really the most interesting thing in the Panama Papers touching the Clintons?

If I'm reading this right, there's a well-known group of lobbyists/Washington insiders (the Podestra brothers) with ties to the Clintons and Obama, and the new revelation is that the lobbying firm is also retained by Russia's biggest bank? (As opposed to, being retained by the US's biggest bank?)

Does anyone else find that weak-sauce?

<comment><USER3> They don't just have ties, John Podesta is Clinton's campaign chairman. And the Podestas aren't just lobbying for Russia's biggest bank. As the article states, the majority shareholder in Sberbank is the Russian Central Bank. From the Wikipedia page on Sberbank of Russia ([https://en.wikipedia.org/wiki/Sberbank_of_Russia#Ownership](https://en.wikipedia.org/wiki/Sberbank_of_Russia#Ownership)):

"The majority shareholder of Sberbank is the Central Bank of the Russian Federation, owning 50%+1 voting share of Sberbank's voting shares."

Effectively, the Russian Central Bank controls Sberbank, but ostensibly Sberbank is a private institution. The Podesta Group registered as a lobbyist for Sberbank, which effectively gives the Russian Central Bank a lobbyist in Washington. If the Russian Central Bank had registered directly with the Podesta Group, as the article states the Podesta Group would have to register as agents of the Russian government under the Foreign Agents Registration Act.

The wikipedia page on the Foreign Agents Registration Act ([https://en.wikipedia.org/wiki/Foreign_Agents_Registration_Ac...](https://en.wikipedia.org/wiki/Foreign_Agents_Registration_Act)) states:

"...[requires] that agents representing the interests of foreign powers in a "political or quasi-political capacity" disclose their relationship with the foreign government and information about related activities and finances. The purpose is to facilitate "evaluation by the government and the American people of the statements and activities of such persons.""

Arguably, the Podesta Group is not facilitating evaluation of statements its making on behalf of Sberbank, and by extension the Russian Central Bank, because it registered as a lobbyist for a bank controlled by a state-owned bank. The Russians suffered heavy sanctions recently. It doesn't stretch the imagination that the Podesta Group received money from Russia to lobby Washington to drop those sanctions.

As the article states towards the end, if Clinton wins the election, John Podesta stands to gain a position of power in a Clinton administration. If John Podesta benefited from money provided by a foreign government (again, Sberbank is controlled by the Russian Central Bank), then that's a relationship the Clinton campaign needs to explain.

<comment><USER4> What is the revelation here though? It seems to me that there are two primary data points in this article:

      - Podesta group lobbies for Sberbank
      - Sberbank is owned by the Russian government

Neither of these things were revealed by the Panama papers, and neither of these things were hidden by any of the parties involved. Sberbank is transparent about its ownership by the Russian government, and the Podesta Group registered as its agent of its own accord prior to the leak.

If this were going on in the shadows, maybe there'd be something here, and I grant you that it's slightly unsettling that they have access to such a powerful lobbying firm. But isn't that sort of the point of running a lobbying firm? You lobby for who hires you, and the politicians you lobby are aware of that and take what you have to say with the appropriate grains of salt.

<comment><USER5> > Neither of these things were revealed by the Panama papers

are you sure? The article says that

      Almost lost among the many revelations is the fact that Russia's
      biggest bank uses The Podesta Group as its lobbyist in Washington, D.C.

so I wonder if you have a reference to back your statement?

Also, you say

> Sberbank is owned by the Russian government

but this is not the point of the article. They are saying that Sberbank is _controlled_ by the Russian government, which is subtly different.

<comment><USER4> From the article: "As reported at the end of March, the Podesta Group registered with the U.S. Government as a lobbyist for Sberbank, as required by law"

And are those things different? Maybe Russian corporate law is different than it is in the US, but owning 51% of a company generally gives you control of that company, unless there are weird share structures like non-voting shares a la Google. But generally speaking, majority ownership and control are synonymous.

</comment></comment></comment></comment></comment><comment><USER6> This is the biggest non-story I've seen from the Panama Papers. There are a lot of revelations in there, but this has no place in the news at all. A lobbyist group connected to the Clintons is also connected to a Russian Bank. They're a lobbyist group. They take money to lobby the government from many different people. That's their purpose.

Total non-story trying to sound like a huge deal, though.

<comment><USER7> Everyone else here seems to hear the words "big bank", and think Sberbank must be sort of like Wells Fargo. But the thing is, Sberbank just isn't any old "bank":

 _Funds moving through Sberbank are regularly used to support clandestine Russian intelligence operations, while the bank uses its offices abroad as cover for the Russian Foreign Intelligence Service or SVR. A NATO counterintelligence official explained that Sberbank, which has outposts in almost two dozen foreign countries, "functions as a sort of arm of the SVR outside Russia, especially because many of its senior employees are 'former' Russian intelligence officers." Inside the country, Sberbank has an equally cosy relationship with the Federal Security Service or FSB, Russia's powerful domestic intelligence agency._

Which make the question of why the Podestas would choose to take their business a very interesting question indeed.

<comment><USER8> Just wait until you hear what the cia gets up to. Or the state dept.

I think your surprise all over this thread is due to people not being stunned that water is wet, politicians need money, and associated hangers-on are venal.

Hell, look at the ties between Bush and the Saudi theocrats, whose greatest hits include 9/11 and are currently doing their best to drag us into their Iran proxy war.

<comment><USER7> I wasn't saying that I was conceptually surprised by the connection. I was just providing background for those who seemed not to appreciate its importance.

</comment></comment></comment></comment><comment><USER9> Is this a sentence or just a collection of words:

Sberbank (Savings Bank in Russian) engaged the Podesta Group to help its public image—leading Moscow financial institutions not exactly being known for their propriety and wholesomeness—and specifically to help lift some of the pain of sanctions placed on Russia in the aftermath of the Kremlin's aggression against Ukraine, which has caused real pain to the country's hard- hit financial sector.

<comment><USER10> I ended up reading that sentence four or five times before I finally interpreted every word and was able to parse it. I kept accidentally skipping words that changed how it parsed.

<comment><USER11> I found it pretty clear... just ignore the bit in the em dashes:

> Sberbank (Savings Bank in Russian) engaged the Podesta Group to help its > public image and specifically to help lift some of the pain of sanctions > placed on Russia in the aftermath of the Kremlin's aggression against > Ukraine, which has caused real pain to the country's hard-hit financial > sector.

<comment><USER10> I kept getting tripped on on some specific word being missing in the aside, but I can't identify what it was anymore. It's less that it was "just a collection of words" and more that I'm fairly sensitive to reading grammatical errors (even if I'm often rushing too much to prevent them in my own comments), and I kept interpreting one where there wasn't one. That's not necessarily supporting evidence for the original comment, as even though I did remember having some problem with that sentence specifically, the fault could easily have been with me.

</comment></comment></comment></comment><comment><USER12> The author of this piece (John R. Schindler) is a pretty interesting person:

[http://20committee.com/about/](http://20committee.com/about/)

<comment><USER13> _" Admit nothing; deny everything; make counter-accusations."_

Man is obviously insane, needs urgent help.

;-)

</comment></comment><comment><USER14> Seem to be 2 camps here: 1\. dismiss this as a tenuous connection which stems mainly from the fact that Lobbying firms have many big clients 2\. grant this connection deserves some attention and perhaps some explanation

For me, I think the author has some thoughts that are bleeding through . But I don't think its necessarily fair to judge an article mainly on its author's history (though an undeniable factor).

While granting this is the nature of a lobby firm, it does seem odd they would agree to representing Sberbank. We're talking John Podesta (Clinton campaign chairman) -> Tony Podesta (CEO Podesta Group) -> Sberbank (Podesta Group client). Is the latter connection really not cause for suspicion?

<comment><USER2> The tricky thing is that it _could_ be a cause for _suspicion_ , but you could also read it other ways.

Before John Podestra was the campaign chairman for Hillary Clinton, he was the Counselor to Barack Obama. Before that, he was the founder of the Center for American Progress (a left-wing think tank). Before that, Bill Clinton's Chief of Staff. Before that, Deputy Chief of Staff, Assistant to the President, and a senior advisor.

Which is to say, he has been a big part of the Democratic Party / progressive movement for _a long time_.

And before that, he co-founded a lobbying group, the Podesta Group, with his brother Tony Podesta, who still runs the Group.

So, do I think that a giant bank / nefarious Russian secret agency is likely retaining the Podesta Group because it was founded and still run by the brother of a powerful and influential member of the Democratic Party, who has served in several Democratic administrations and is likely to serve in future ones? Yeah, totally.

And it's probably worth being aware of the potential chain of influence and corruption -- client to lobbyist, lobbyist to advisor, advisor to politician -- but I find it far more likely that John Podesta is in Hillary's campaign (and potentially, future administration) because he's a good, useful guy who gets the right stuff done, rather than that he's there as some sort of corruption-conduit, proof (PROOF!) of Hillary Clinton's unworthiness.

</comment></comment><comment><USER15> This seems like a pretty tenuous 'connection'.

<comment><USER7> Sberbank is very close to the Kremlin. The connection could scarcely be _less_ tenuous.

As the article itself asks, the main question is why one of Clinton's chief power brokers chose to take their money.

<comment><USER16> none of this information was a secret before the Panama leak

</comment></comment></comment><comment><USER17> The Clintons use Windows, The Russians use Windows.

"Microsoft's Financial Reports Reveal Clinton's Kremlin Connection"

</comment><comment><USER18> So the choice is between Putin's favorite Trump and now apparently Clinton as well? Though Bernie would probably practice appeasement toward Putin too. Can't you Americans elect a strong leader for once?

<comment><USER19> You want another Bush? We have another Bush.

<comment><USER18> He dropped out unfortunately. You guys may hate GWB, but he sure kept Putin in check. I think he used to call him something like "pooty-poo", in person!

<comment><USER20> A collapsed economy kept Putin 'in check'.

Oh, and have you forgotten about the Russian-Georgian conflict over South Ossetia? [1]

[1] [https://en.wikipedia.org/wiki/Russo- Georgian_War](https://en.wikipedia.org/wiki/Russo-Georgian_War)

That one was on Dubya's watch - whereby he proceeded to do nothing. It's pretty clear to everyone that Great Powers will do as they please in 'their' sphere of influence.

If Canada and Mexico were on the verge of joining a military alliance with China, there would be Abrams tanks in Ottawa, and F-22s bringing democracy to Mexico City before the week would be out.

<comment><USER18> Did Putin annex South Ossetia? He didn't. Georgia returned to status quo. The reaction of the West was completely different back then.

<comment><USER20> South Ossetia is for all intents and purposes, a vassal state. The distinction is academic.

Georgia did not return to status quo - it lost territory. (If the NATO/UN position on this is to be believed)

The reaction of the West was quite different - it largely sat back, and watched.

</comment></comment></comment></comment></comment></comment><comment><USER21> Woohoo, the Sanders people found some lobbyists that have a connection with a foreign bank that happen to be connected to the Clinton's. By their logic, this proves Clinton is a pawn of Vladamir Putin.

This has got to be about the least interesting thing to come out of the Panama Papers.

</comment><comment><USER22> Yes, I'm pretty sure that's what Americans want - a big scandal around Clinton that would land Trumpet in charge.

<comment><USER23> That is a false choice that the Clinton campaign would love for you to believe. In actuality, the FBI investigation alone would have derailed any other candidate, now this... For all the talk about Trump being teflon, it sure seems Hillary has the some of the same "teflon-ness"

Sanders has won 7 out of the last 8 primaries and yet Clinton, with all her legal baggage and ethical issues, is still the presumed frontrunner.

<comment><USER24> I was talking about this with a friend last night. Think of all the scandals the Clintons have endured - from Whitewater to Travelgate to Lewinsky to Benghazi...and yet nothing has impeded them. That either means they are in fact predominantly innocent (I think we can rule this out, mostly...) or that they are just such masters of politic and influence that they are able to fend off any investigations into them. Seriously, Clinton lied about cheating on his wife on national television and yet people still love him. They are a very intriguing couple. I eagerly await a Caro-esque look at their role in US history.</comment></comment></comment>


Why I'm Worried About the Repo Market - <AUTHOR> https://www.bloomberg.com/opinion/articles/2019-09-25/why-i-m-worried-about-the-repo-market <comment><USER2> It's essentially QE4. Fed's balance sheet is increasing (even if for a short duration). Make no mistake - this results in lower return on cash for savers. At a time when big banks call 0.05% "high yield", this is the last thing we need. If banks are short on reserves they should be increasing interest rates to attract new deposits and maybe selling some of those foreclosed homes they've been holding onto for an eternity.

The banks have completely failed as a business and the fed is enabling their incompetence. How many stories do we need of money laundering, market manipulation, fraud, until we say enough?

The best part of all this? We don't even get to know the real reason there's a liquidity crisis. Obviously the "it's investors buying treasuries and corporations paying taxes!" explanation was BS.

<comment><USER3> Who are you referring to when you refer to "savers"? Interest rates have been basically 0 for a very long time for retail bank customers.

<comment><USER4> Have been getting 1% monthly uncapped for almost 8 years now on my checking account.

<comment><USER3> That is an exception, not the rule. If you have Chase, Citi, BofA, etc. it's been like .01% for 10 years.

<comment><USER5> If you're willing to move your funds to another institution, check out [https://www.depositaccounts.com](https://www.depositaccounts.com) to see how much you could be earning with your money.

</comment></comment></comment></comment></comment><comment><USER6> The Fed created this problem. Yet he wonders why.

It's reasons like this that cause people like Ron Paul to think the Fed is dangerous - they don't understand their own creations.

Kocherlakota doesn't understand it either.

The repo market explanation is simple: the IOER rate is higher than fed funds, but with the Fed reducing their balance sheet (QT) by $800 billion, there are going to be unusual effects. This is one of them.

<comment><USER6> For anyone who would like further reading on this, there is a great (readable) article here:

[https://www.advisorperspectives.com/articles/2019/09/25/unde...](https://www.advisorperspectives.com/articles/2019/09/25/understanding- the-great-repo-fiasco)

</comment></comment><comment> 33MHz-i486 Without Fed intervention, the repo market is just big banks lending to each other overnight, using mostly risk-free assets as collateral.

Why wouldn't this market clear at a large spread above the fed funds rate? ... either a majority of banks are too low on capital reserves to participate in some lucrative low-risk loans, OR one or more unknown participants are close to insolvent and the others don't want counterparty exposure. Either way it's a dangerously close to a systemic liquidity crisis and the Fed is completely behind the ball.

</comment><comment><USER7> Oh it's the _regulation_ that's the problem, not the inability of banks to not overleverage themselves.

<comment><USER8> Overleverage == self regulation

</comment></comment><comment><USER9> The reason why we start to see problems in the repo market is clear: the FED is paying banks to keep their money in a safe place, being rewarded for taking no risk! Why would a bank ever lend money to others if they see any problem whatsoever appearing in the horizon? In the next recession, you will see that ALL banks will stop lending to each other and let their excess reserves safe and sound receiving interest for doing nothing. The FED has created the very conditions for a sure repetition of what happened in 2008.

</comment><comment><USER10> That's only one consequence of the regulations taken after 2008 (the leverage capital requirements forcing banks to shrink their repo book, reducing liquidity in the repo market).

Tighter rules on capital requirements for holding many other asset classes, bonds in particular will also reduce a lot the capacity of banks to make a market for these instruments in bad times, resulting in more volatility.

None of that is new, banks have been warning about that for years. But it will only become visible in a sharp market downturn.

</comment><comment><USER11> Noob question: does this have something to do with inverted yield curve? Isn't it simply more useful/profitable for banks to finance their operations with short term debt right now?

</comment><comment><USER8> This article is unreadable on mobile with popups and scroll injection.

<comment><USER12> Looks fine on my iPhone.

</comment></comment><comment><USER13> Isn't this just a short-term shortage of credit? We have experienced similar things on different levels many times. Or did I miss any thing?

</comment><comment><USER14> Outline to the rescue [https://outline.com/DuqfDW](https://outline.com/DuqfDW)

</comment><comment><USER15> Let banks fail? Arrest the execs and bankers who can't keep enough reserves?

<comment><USER16> Arresting someone is always a good thing to do. They tried it in the Soviet Union in 1937. Worked like a charm.

I have a question though: do we arrest everyone who fail to perform, or only bankers?

For example, our county recently had to pay 16 million to the family of the guy who was killed because of a faulty traffic light. Should our county executive go to prison?

Also my train was late this morning by 40 minutes due to what they called "equipment failure". Should a person responsible for maintaining the equipment be arrested as well, or execution in front of the passengers would be a better measure here?

<comment><USER17> There's a difference between failing to perform and specifically making choices that hurt others.

If the county executive knowingly chose faulty traffic lights, yeah they should go to prison.

The bankers aren't bumfuzzled or victims of circumstance, they're just not optimizing for public good or the stability of the system, because why should they? They have no incentive to. If shit goes sideways, they'll just get bailed out.

So how do we change the incentives? Not sure personal liability is the answer, but it's not the worst idea in at least some cases. (Though in this specific case, as much as I even understand what's going on, the answer is probably just standard regulation on the banks themselves)

<comment><USER5> > The bankers aren't bumfuzzled or victims of circumstance, they're just not > optimizing for public good or the stability of the system

Even with all the nonidealities of the USA financial system, it is still pretty darn stable. If you've visit other parts of the world, it is apparent that the USA financial system (and political system, notwithstanding media reports) is among the most stable in the world. This is why companies around the world seek financing in the USA and want to float their shares on NYSE and NASDAQ.

Is there room for improvement? Absolutely.

<comment><USER18> It's stable until it's not.

</comment></comment></comment></comment></comment><comment><USER19> Just want to plug a bank I love that is founded on good principles.

aspiration.com

5 mins to open an account with a bank which has a great "social conscience" and truly great perks you won't get from other "big" banks.

No. Seriously, take a look.

Also, in case you are wondering: your credit is not touched when signing up, so really - it's a no brainier

You can thank me later.</comment>