A New Breed of Trader on Wall Street: Coders with a Ph.D - <AUTHOR> http://www.nytimes.com/2016/02/23/business/dealbook/a-new-breed-of-trader-on-wall-street-coders-with-a-phd.html?src=busln <comment><USER2> Lot's of "nothing new" comments here, but I think that may be missing the point.

Wall Street has been hiring physics & math graduates for a good while now, but typically as quants, not traders. Those were (are?) hugely different positions, in terms of responsibility and dollars realized.

Not my area, so I may just be missing something too. But when I had acquaintances going into this sort of things, they weren't getting anywhere near a trading desk. At least, not for a while.

<comment><USER3> The line between trader and quant in electronic trading scenarios is pretty vague and has been for some time.

<comment><USER4> One gets you fired when you lose money and the other gets you a speaking opportunities at universities where kids are discovering Gordon Gekko.

<comment><USER5> You want it to be that way, but it's the other way.

If you're just a "cog in the machine" doing research or writing software for a trader, and he ends up sucking, the firm will probably shuffle you around to work under someone else. You have generally useful technical skills, cost less than the trader, and recruiting good technical people is expensive.

If you're a supposedly brilliant trader but cost the firm money taking stupid risk or fail to make any profit to justify your inflated salary and bonus, there's not really much they can do with you.

<comment><USER6> .... like john merriweather or victor niederhoffer for instance. two guys who've blown up the world multiple times but still manage to find capital and manage huge funds.

</comment></comment></comment></comment></comment><comment><USER7> Wait, the article cites Knight Capital with a straight face and doesn't bring up the whole "Whoops we lost $440 million due to bad software" [1] as a significant caveat to putting so much emphasis on the role of technology? That doesn't seem very contextually fair. Programmers, no matter how great, can still make costly mistakes.

[1] [http://www.bloomberg.com/news/articles/2012-10-17/knight- cap...](http://www.bloomberg.com/news/articles/2012-10-17/knight-capital- reports-net-loss-as-software-error-takes-toll-1-)

<comment><USER3> The Knight Capital responsible for the Knightpocalypse basically went out of business due to it, and was bought by Getco a different trading firm. Getco switched names because in some places the The Knight brand was stronger than the Getco one.

<comment><USER8> They actually did this to reverse-IPO, giving all of the PE firms and partners a way to get liquidity since KCG was already public.

<comment><USER3> Do you know how they ended up splitting up the ownership stakes? I've been curious but never tracked it down?

</comment></comment></comment></comment><comment><USER9> I've been in and out of coding for finance at a few different shops for about 10 years now, and recently have been working more on the front-lines: coming up with my own trades and the software to do those. I can tell you from personal experience that this is a _completely_ different experience than writing code for someone else (a trader) who then is responsible for making/losing the money.

Imagine this for a daily routine: 1) oh shit this is awesome: x times $1000000 means i'm going to be so fucking rich! 2) run code live on the XYZ market, trying not to shit bricks 3) hang-on i found a bug in my analysis/data- preprocessing/ML pipeline and i'm not going to be rich. 4) spend hours (or days) fixing bug & re-running analysis 5) go back to 1).

After a few months of this one gets seriously fragged.

<comment><USER10> ha ha, I did that basically with bitcoin.

</comment></comment><comment><USER11> I occasionally meet bankers here in London and I can't help but think to myself what a waste of time and effort when there are actual problems to solve. Bit rude I guess but a whole generation of bright people reducing the latency of trades or compliance software or a new trading algorithm does not move us from zero to one, maybe more zero to -0.3 or something.

How people can claim the market allocates money correctly I do not know, it doesn't seem possible with the rigging and illegal activity that's been going on.

<comment><USER12> I think it is odd to conflate "bright people reducing the latency of trades" with "rigging and illegal activity". It seems to me that the former are rarely involved in the latter. One only needs to look at the libor cases, the FX fixing cases, or the gold fixing case.

I actually have a pet conspiracy theory that the articles about HFT are the result of lobbying on the part of banks. When I worked at a bank, I did see articles that were heavily influenced by employees of the bank. This influence was never nefarious, but I can see how a traditional trader's biases may have lead to some of these HFT articles.

I rarely see articles about the evils of internalization, something that I strongly believe brings great harm to the markets. At the same time, I often see articles about how HFT takes advantage of the market and brings little benefit when I've made internal measurements that show just the opposite.

Of course none of the above is any sort of advice, and it's probably appropriate for me to note that I've worked as trader at a bank and currently work as a trader at an HFT firm.

<comment><USER13> > I rarely see articles about the evils of internalization, something that I > strongly believe brings great harm to the markets

How?

<comment><USER12> Internalization disincetivizes people from making tight markets in lit venues because you will only get filled when an internalizer thought the flow was toxic. This in turn means that markets are wider for all participants.

</comment></comment></comment></comment><comment><USER14> What's new is that they are using FP instead of C++ to deliver results quickly and more reliably.

These aren't the quants from the 90s that were Physics PhDs applying the latest and greatest in differential equations and whatnot. These are programmers who know how to make the complier do more work for them so they achieve results faster.

<comment><USER15> Fast execution with very terse code, and also the code that is produced by a diciplined OCaml programmer can be easier to grok in many situations because it may read close to what the math formulas look like on paper relative to other languages.

To be honest this is the same argument people make for Haskell and I'm only assuming the argument applies to OCaml equally well.

</comment></comment><comment><USER16> Trader+coder here.

This article is missing something very important, the fact that coding doesn't mean you're going to trade better.

A coder doesn't know how to trade right off the bat, they'll only know how to code in a specific trading rubric (most trading algos aren't even good unless we're talking HFT) that has been told to them from someone that already knows how to trade successfully.

Anyone that's a real trader and has managed clients money will instantly be able to tell you that this is articles title is misleading.

<comment><USER6> a "real trader" manages his / her own money, not that of clients.

<comment><USER17> the street doesn't make that distinction

</comment></comment></comment><comment><USER15> I wonder if their recruiting tactics are as aggressive as some of the firms I've seen portreyed in film..

MARCY DAWSON: I admit we've been too aggressive. But all I ask is that you give me five minutes. As a token, accept this.

MAX: I don't want your money

MARCY DAWSON: The suitcase isn't filled with dollars, or gold, or diamonds. Just silicon. A Ming Mecca chip.

MAX: Ming Mecca? They're not even de-classified yet.

MARCY DAWSON: You're right, they're not. But Lancet-Percy has many friends. Beautiful, isn't it? Do you know how rare these are?

MAX: What do you...?

MARCY DAWSON: Mr Cohen?

 _MAX RUNS DOWN INTO A SUBWAY_

(taken from π screenplay by Darren Aronofsky)

<comment><USER18> I don't deal with petty materialists like you!

One of my favorite movies.

</comment></comment><comment><USER19> I'm currently enrolled in a MSc program with a thesis on portfolio optimization using statistical learning. It might even lead to a publication! But now, the question is, should I go onto a PhD in stats or CS or just settle with it. I know I probably have the chops to complete it, it's just that 4 or 5 five more years in school is somewhat risky. Sometimes it's hard to assert your worth on such a competitive market.

On a side note: omg janestreet is such my dream job. Writing functional code, working on immensely interesting problems (what is the market distribution?) and being paid for it.

<comment><USER20> I'm not sure I understand these jobs in an economic sense and I still am not sure of the value they provide to society. Does solving liquidity or arbitrage issues a few microseconds, or even seconds, faster than someone else improve productivity? Or when it doesn't depend on speed but on seeing some tiny market inefficiency, how much does that help the world? (And I'm sure I'm overlooking other economic activities in these firms - what are they?)

There's the economic question of why the market rewards them so well (not that, in the real world, the reward is usually proportional to the value, but it's something to think about). Is it essentially the equivalent of rent- seeking - that is, they get their hands on the resource first and then sell it to others? Or perhaps it's simply the free market, but working on a timescale of microseconds, where efficieny does't really benefit a society of humans that live on timescales orders of magnitude larger.

Finally, there's the concentration of wealth that results from these activities, which is a serious concern.

(I'm not saying everything everyone does must contribute to society, or that I'm a saint who lives on nuts and berries and tends to lepers, but it's a consideration.)

<comment><USER21> This is the crux of the matter. No, there's no evidence that these activities produce economic value of a magnitude that justifies their compensation.

A certain number of "speculators" are needed to provide liquidity to capital markets. It's unclear what that number is, but it is most likely nowhere near the level we currently see.

In essence, it is gambling. These funds are supported on an ongoing basis by asset management fees. This provides a nice living for the fund managers and employees. They are structured so that the players experience all upside and no downside. If they lose millions of dollars it's not like the employees and managers are responsible for making it up (although sometimes there are clawback provisions, but not nearly as often as you would want). But if they make millions they take a cut of their gains.

This most likely leads to unnecessary risk-taking. (unnecessary in the sense that there's no net economic benefit from it).

So if this is largely a zero-sum game, who are the losers and why don't they do anything about it? Good question. Most directly it is the institutions and wealthy individuals who invest with hedge funds.

These institutions/individuals do it because as a species we have a flawed reasoning capacity and overestimate our ability to exercise good judgment.

<comment><USER22> >No, there's no evidence that these activities produce economic value of a magnitude that justifies their compensation.

The evidence of economic value is the fact that these activities are so lucrative. If they weren't producing a lot of value somehow, be it directly or indirectly, they wouldn't receive such high compensation in the market. Economic value is essentially a measure of what people are willing to pay for things (revealed preference), under the assumption that people paying for something, absent any cooercion or market distortion, means they value it.

<comment><USER20> How much did Albert Einstein make compared to Babe Ruth? Why is the highest paid employee at most large public universities not the Nobel Prize winning scientist, but the football coach?

Even in theory, many conditions are required for free markets to function efficiently. Regardless, we don't live in a free market, and many other factors determine income.

<comment><USER22> There were far fewer people willing to pay to see Einstein do what he did than to see Babe Ruth did what he did. Millions of people got regular satisfaction from watching Babe Ruth, which suggests more people valued what he did than they valued what Einstein did. Economically speaking the value an average person places on something is no less valid than the value an intellectual places on something, and there are a lot more average people than intellectuals.

</comment></comment></comment></comment></comment></comment><comment><USER23> Title mentions PhD yet only 1 of the 6 guys in the article has one.

</comment><comment><USER24> The first three comments were all variations on "this isn't new" so let's take that point for granted, and see what else this article gives us.

It's not new that Wall Street is hiring Ph.D scientists and mathematicians (that's been happening since the 80s) but what's changing is the _kind_ of role they are getting hired for.

From the 1980s up until 2005 or perhaps even later, most Ph.Ds were hired in "quant" roles, that is, to build mathematical models that could price and manage the risk of derivatives. They were generally not in trading roles.

More recently (i.e. in the last decade) it's common to hire Ph.Ds as traders - that is, to write algorithms that are used to make trading decisions. This requires a rather different set of skills - instead of being skilled in stochastic calculus, partial differential equations and numerical methods, the quant trader is skilled in statistics, data analysis, optimisation and machine learning.

It's true that Wall Street has Ph.Ds hired to build trading models for many years (e.g. Renaissance) but the change is that this is no longer an esoteric fringe pursuit. It is seen as standard that trading desks at investment banks will have a large number of quant traders. Large European investment banks have heads of trading who are quants. Quantitative hedge funds are no longer mysterious and exciting - many of them use well understood models that have been widely replicated across the industry.

To be honest, I am surprised that it has taken this long. Finance is so obviously suited to these kinds of quantitative methods that only an aggressive rearguard action by voice traders has been able to keep them at bay. The question is no longer whether quantitative traders and their algorithms will largely replace voice traders, but how long the voice traders will manage to hold out.

Jane Street is an interesting special case. From my (somewhat limited) interactions with them, they are neither wholly voice traders nor wholly algorithmic. Instead, their researchers and traders build algorithmic trading systems which can be "driven "by humans. For example, the system will continually calculate a set of useful metrics that a human trader uses to make a final buy/sell decision, and trade is then immediately executed by high frequency execution algorithms. Or the human trader decides to make a complicated spread bet (e.g. long an ETF vs. a cost-optimized basket of the underlying stocks) and the algorithm goes out to execute the basket as effectively as possible.

I think there's an interesting parallel to "Centaur chess" [0] where the combination of a computer and a human is much more powerful than either of them acting alone.

[0] [https://en.wikipedia.org/wiki/Advanced_Chess](https://en.wikipedia.org/wiki/Advanced_Chess)

<comment><USER25> That's the essence of complementarity. Cowen's "Average is Over" talks about it a lot, and uses many chess examples specifically to back it up. Recommended.

<comment><USER26> I thought the book was alright but spent way, waaaaay too much time on chess.

</comment></comment></comment><comment><USER27> I'm coming to the end of my PhD, and have been contacted by a few recruiters for various hedge funds. It seems like a shame to leave academia. Realistically though, most people will be stuck in a series of ~18 month postdoc contracts, with little prospect of getting tenure. Add to this the poor pay, and hedge funds start to seem pretty attractive!

</comment><comment><USER28> This industry irks me.

What's the point of doing engineering anywhere if you don't get paid as much as these guys to shift money around?

<comment><USER29> The pay in these sorts of articles is always overhyped. As with any business, the people making the most money at these kinds of firms are the owners of the firms. Consider this line

 _"As a trader, if you do well, you will retire before you turn 30," said one employee on an industry message board._

That is basically 100% unattributed, unconfirmed bullshit.

<comment><USER5> Yeah that sounds like complete bullshit to me. I know some top quant traders who make high-six/low-seven figure comp consistently at that point in their careers, but they're nowhere near able to retire with a comfortable lifestyle, and that's an extremely biased sample. For every person like them, 100 others fail early on, 10 stick around making a nice but not much higher than Google salary, and 5 have a couple busy high comp years then get pushed out by greedy management, have their edge eaten by competitors, or quit when the markets slow down.

Getting to that level is like making partner in a law firm. You have to do much better than "well" and anyone who gets there would be a fool to retire and likely wouldn't want to anyway due to the personality traits/defects required to get there in the first place.

</comment></comment></comment><comment><USER30> Interested to find out why OCaml. Not that it's wrong in my view, but why, next to so many choices.

<comment><USER31> I imagine it's because until about 5 years ago, there weren't many choices that were significantly safer than Java (no null references, failing to account for all cases in a match statement is a compile error, etc.) and generally within a factor of 2 of C's speed. Haskell's laziness by default can mean less predictable performance, or at least fatter tails on the performance distribution.

<comment><USER32> What would be the choices now? Most references to alternatives suggest Haskell.

</comment></comment></comment><comment><USER33> This seems to be the wall street equivalent of transitioning sysadmin roles into devops.

</comment><comment><USER34> "Writing computer code, or at the least being conversant in the firm's program of choice, OCaml, is a requisite for all traders. Indeed, new traders must complete a monthlong OCaml boot camp before they start trading."

</comment><comment><USER35> Reads like a recruitment piece for programmers, or maybe that's how I read too many articles being programmer. I was a little confused by this:

> For example, Jane Street, which is privately held, has increased its > shareholder's equity, or net worth, to more than $1 billion today from $228 > million in 2007.

Is this supposed to be an impressive stat? ZocDoc has become about $2 billion company in basically the same time frame, which seems more impressive given that ZocDoc didn't start from that $228M handicap. My perception is that it's also much more expensive to run a company like Jane Street.

<comment><USER36> you are confusing valuation with shareholder's equity

<comment><USER35> Okay. Thank you. This is probably the source of the confusion. Would a company like ZocDoc even have shareholder's equity?

<comment><USER37> Of course, shareholder equity is just assets - liabilities.

Granted, it's probably significantly smaller than the $2B figure you cited.

</comment></comment></comment></comment><comment><USER38> If there's one place where just about everyone I know has done job interviews, it's jane street. They cast a very wide net.

<comment><USER39> I see their advertisements everywhere. Probably because I searched OCaml at some point.

</comment></comment><comment><USER40> As a swing trader, this interests me. The volatility of the market presents opportunities I don't think would otherwise be possible. Certain news items will push a stock significantly below market value, as determined by 10-Qs overtime (and _some_ knowledge of a given industry). I happen to believe the current AI-led trading induces extra volatility in the market that is eventually corrected for which ultimately leads to swing trades.

</comment><comment><USER41> Not new. A high school (early 2000s) physics teacher told us all that if we wanted to make money with physics, we should:

1\. Get a physics PhD with

2\. Some experience in computer modeling

3\. Get hired by Wall Street

Even with the downturn in '08, I've seen no indication he was wrong.

<comment><USER42> Yeah, half my friends from university graduated straight out of their Physics degree and went straight into finance. Didn't even bother with the Masters or PhD.

</comment></comment><comment><USER43> To all the "Nothing new here" people - really?

Yes there is nothing new, in that you physically cannot write the words "Jane Street Capital" without subsequently writing the the word "OCaml". But seriously, an OCaml reference in the _New York Times_. Someone must be celebrating.

</comment><comment><USER4> 1986,87 also saw a boom in quants, mainly engineers, even nobel prize laureates. They put them in one room, come up with ways to make money using quantitative models. There was a lot of buzz for decades until the collapse of LTCM.

Peter Thiel said MBAs flocking to certain tech as warning signs of a peak, likewise wall streets have had PHDs flocking as a measure of it's peak.

I fear that with a double whammy of low oil prices and low commodity prices, it will spark a long period of economic stagnation in the near future, as more jobs disappear in increasing numbers and social benefit falls and governments around the world struggle to monetize their aging populations.

The countries with young and increasing birth rate will feel the impact the worst, and we are already seeing such demographic in Europe packing up their bags and joining jihadi movements.

<comment><USER44> > I fear that with a double whammy of low oil prices and low commodity prices, > it will spark a long period of economic stagnation in the near future, as > more jobs disappear

So if commodity prices go up, it's bad because Europe can't afford those commodities. But then if they go down it's also but because...?

Heck, it's not a double whammy, it's good news for most countries in the world. A different story might be _why_ are these commodities going down (because China's demand has been drying up), and that might actually be bad news, but it's another argument.

</comment></comment><comment><USER45> I think wall street has been hiring PhDs to code for a long time. I don't think this is a new phenomenon.

<comment><USER46> Yea isn't the old joke that quants are composed of Physics PHD dropouts ?

<comment><USER47> Here is the breakdown for the quants at the bank I worked for in 1995-1999 (I may have forgotten a few people):

Cornell physics PhDs: 3

Harvard physics PhDs: 2

Princeton physics PhDs: 2

Univ of Penn physics PhD: 1

Other physics PhDs: 2

Cornell applied math PhD: 1

Other applied math PhD: 1

Cornell mechanical engineering PhD: 1

Univ of Chicago finance PhD: 1

So, if by "dropout" you mean that they didn't complete their PhDs, then no, that's wrong. If by "dropout" you mean people leaving physics, then yes, it's very common (especially for theorists -- all physicists on the list above where theorists except maybe one or two).

<comment><USER48> On the assumption that their dissertations were not on relevant topics, I can't help but feel this is such a terrible waste of highly specialized education resources.

<comment><USER49> Are there lots of unfilled physicist jobs where you are?

<comment><USER48> Yes, very snarky, but I can promise you that science faculty do not consider it a good use of their efforts and resources to train students for jobs moving money around in circles.

<comment><USER49> We get to train them, not choose what they do afterwards.

<comment><USER48> That's not in dispute. I'm saying it's a waste, not that I know how to fix it. More relevant jobs is obviously a part-answer.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER50> Nothing remotely new

</comment><comment><USER51> How are people bypassing NYT and WSJ paywalls these days? The Innocuous Chrome Extension posted a few days ago on HN no longer works, and coming from Google isn't working for me either.

<comment><USER52> This may surprise you, but some of us actually have subscriptions. I don't have one to the WSJ but I have one to nyt.

<comment><USER53> What made you subscribe to the Times versus other newspapers? I've been debating a Times vs. WSJ subscription.

<comment><USER20> Some differences, from someone who knows far more about news and the news industry than they should (and I'm not in the industry):

* Very generally, the Times focuses more on investigative journalism and non-business news; the WSJ's priority is business news. But they both cover plenty in all fields.

* The Times is generally a little more prominent, the leading newspaper (and news source) in the world and the "Newspaper of Record"[1] for the U.S., but I'm not sure how that reputation benefits you, the reader.

* The Times' arts coverage is unquestionably the best and most sophisticated of any newspaper, if that matters to you, from film to theater to music to exhibitions to other 'fine' arts. However, it is somewhat New York-centric (it is the _New York_ Times, after all, and NY is the center of the U.S. and global art world).

* Editorial: Even the best newspapers' standards of accuracy on their editorial pages are far below those of their journalism; they often are happy to print outright lies and propaganda either to push a political issue or because it's influential. IIRC, maybe a decade ago the Times decided they would raise the standards somewhat on their editorial page. Regardless, I find the Times publishes less outright bullsh-t; they have a politically diverse roster of columnists but the unsigned editorials are definitely liberal. The WSJ regularly pushes things like climate denial and other bizarre ideas (that serve the conservative, wealthy busiess elite) such as this one:[2] Personally, I don't read any editorials in any paper - they are so regularly ignorant of or lying about the facts and full of propaganda (that word again) that I never know if I'm becoming more misinformed by reading them.

* Other options: Also, I'd consider the Financial Times, the leading international competitor to the WSJ, which I think is just as good as the other two and provides a more diverse perspective coming from the UK. Certainly there is more overlap in coverage between the NYT and WSJ than between either and the FT - check out their site and you'll see many important stories you'd oterwise miss. The Washington Post also is excellent. I can't think of another source of serious news, in English, in the class of those four.

* IMHO: The WSJ is owned by News Corp (Rupert Murdoch), the same company that owns Fox News. Their demonstrated willingness to lie and push propaganda at Fox News, as well as openly use it to control the political process, makes me doubt that the same owners would have more integrity at WSJ. Their editorial page confirms my doubts to a degree. I don't trust it, though many others do. I don't completely trust the NY Times either, of course, but much more than their Manhatten neighbor.

I hope that helps.

\-----

[1] [https://en.wikipedia.org/wiki/Paper_of_record](https://en.wikipedia.org/wiki/Paper_of_record) \- don't believe some Wikipedia editor's attempt to equate the LA Times and Washington Post with the Times. If any challenges its position, it's the WSJ.

[2] [http://www.huffingtonpost.com/2014/01/30/wsj-defends- kristal...](http://www.huffingtonpost.com/2014/01/30/wsj-defends- kristallnacht_n_4694727.html) \- It describes and links to the original, which is behind a paywall.

<comment><USER54> I would also add that the NYT simply has the best _online_ newspaper. The site itself is beautiful, and I am continually impressed with their innovation in interactive features and design.

<comment><USER20> > I would also add that the NYT simply has the best online newspaper. The site > itself is beautiful, and I am continually impressed with their innovation in > interactive features and design.

Interesting. I would disagree, or at least I think being the "best online newspaper" is sort of like being the best WAP web browser. I think many bloggers do better.

Note that we call it an 'online newspaper' \- often you can instantly identify news websites are made by former newspapers as opposed to other news sources, as if the UI of their website should depend on the archaic medium they once used.

<comment><USER54> What are you thinking of that has both the scope and depth of content of the NYT, and the innovation in design? Or did you just mean one of those two criteria?

<comment><USER20> They're content is unmatched (except arguably by one of the three I mentioned above), I was just talking about design.

As a user, I don't find the innovations very useful and the design functions poorly. For example:

1) Most of their content is hard to discover; the front page must have 100 links to stories, some old and some new, and much more is buried elsewhere - if I wanted to see every story of interest to me, I'm not sure if or how I could do that starting from the home page. I also read it via RSS and it's a whole different news source, with far more coverage. People reading on the web miss a lot.

2) Updates to news stories: If a story has been updated, how do you know? You have to look around the home page for the words 'updated xx:yy' \- not really a great method of notification. And if I click the updated story there is no way find out what's changed without reading the whole story again, looking for things that don't seem familiar.

3) Their inability to integrate multimedia in story-telling: A movie review never says, 'watch the clip below; see how the colors ...' <video> 'also note how the actors ...'. They have images and video, but they are decorations stapled onto the real story in text and not an integral part of it - or even a more dramatic example, the videos with hard news are completely segregated from the text story. They still are a news _paper_ , stuck in the limitations of the old medium where text was the only realistic option, with a few images added on.

Note that bloggers handle many of these issues efficiently. The NYT is still a news _paper_ on the web.

<comment><USER54> These are fair comments, but the NYT also frequently has interactive, data- based visualizations that are among the best in class.

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER55> This is because of the shift to HFT.

<comment><USER56> It's really not, HFT doesn't require quants, it's all about latency arbitrage, not about predicting future prices with quantitative analysis.

It is all about the shift to algorithmic trading, of which HFT is only a small sub-category.

<comment><USER5> Nonsense. Most HFTs are market-makers, either official or unofficial and have to predict price movements to quote competitively. If you only make quotes where you have an immediate arb in another product/exchange, you won't get many trades: [https://ptportal.kcg.com/kresearch/do/research/getDownload?a...](https://ptportal.kcg.com/kresearch/do/research/getDownload?attachmentId=4003&username=8FEW5ecT13JTdHAY7Qm0TA==)

<comment><USER56> HFT market makers aren't doing complex quantative analysis in microseconds it takes to do trades, and latency arb isn't "an immediate arb in another product/exchange", that's just normal arb. Quants are used for longer term stuff, not microsecond level stuff. HTF's are doing order book stuff and flipping the spread, nothing hugely complex, just fast.

<comment><USER5> What is "latency arb" then? To "flip the spread" you need to predict where the market will go after your order gets elected. The market is pretty efficient. The average limit order loses the spread and rebates after execution. You can't make money just being on the bid and offer with the crowd without a directional prediction.

You are right that the actual execution algorithms aren't doing super complex analysis in real-time. Usually that is done offline through simulation to generate rule sets, decision trees or factor weightings. That work is definitely quantitative. Aside from pure arbitrage, most HFT strategies have a statistical edge, not a mechanical one.

<comment><USER56> Latency arb is taking advantage of people with slower connections by being able to place/cancel orders faster than them. As most exchanges are price, time priority, unless someone beats your price, first in line gets filled first.

> To "flip the spread" you need to predict where the market will go after your > order gets elected.

Lets talk abstractly for a moment, that sentence is exactly wrong. The whole point of flipping the spread is that you can immediately sell what you just bought and make the spread risk free. That's what market making is, providing liquidity for the spread as profit. Now, pragmatically, of course the market doesn't sit still for long and the price may change between your buy and sell so of course it's good to predict the market direction to reduce the risk of being on the wrong side of a move. But HFT is trading at the microsecond scale, prices aren't moving much at that scale in that short a period of time and you only need to know where the market is going for say the next second to be in an out of a slew of trades.

> The market is pretty efficient.

Agreed.

> The average limit order loses the spread and rebates after execution.

I don't believe that. That would mean that slippage exceeds the spread and more on most orders and unless you're trading a very illiquid market there's no way that can be true unless you're trading very large, large enough to move the market alone.

> You are right that the actual execution algorithms aren't doing super > complex analysis in real-time.

That's my main point.

> Usually that is done offline through simulation to generate rule sets, > decision trees or factor weightings. That work is definitely quantitative.

I agree with that as well, but that's not HFT trading, that's market analysis and mostly I believe, though I admit I could be totally wrong here, that analysis is mostly used for slower algorithmic trading, not HFT. In HFT, it's better to be faster than smarter, especially with the sub-penny rule imposed by Dodd-Frank.

<comment><USER5> [http://www.tradeworx.com/media/pdf/TWX- SEC-2010.pdf](http://www.tradeworx.com/media/pdf/TWX-SEC-2010.pdf) (p.13) and [https://mechanicalmarkets.files.wordpress.com/2015/01/toa- ex...](https://mechanicalmarkets.files.wordpress.com/2015/01/toa- exchangesummary-13661.png) back up my claim that limit orders lose the spread on average net of costs. In a competitive equilibrium, this has to be the case. If the average limit order lost less than the spread and rebate, it would be profitable to mechanically copy every limit order with an infinitesimally small limit order placed right behind it. If the average limit order lost more, it would be profitable to copy every marketable order trading to remove any residual liquidity on the book at that price. Of course HFTs limit orders perform better than average or they'd be out of business (and many do go out of business).

FWIW, every successful HFT trader I know does this type of "market analysis" driven trading backed by very fast execution. Speed alone is no longer an edge in most markets. Most are holding risk for more than microseconds or single seconds. Trade direction is auto-correlated in short timescales so even being front of the line you are unlikely to buy and sell to earn the spread so quickly.

<comment><USER56> I don't think those don't back up your claim, those numbers are specifically pre-rebate and are being shown there to justify the fairness of rebates as a useful market feature by showing that without them the expected returns would be negative and thus discourage trading. You claimed the average was unprofitable post rebate, showing pre-rebate numbers has no bearing on your claim.

> In a competitive equilibrium, this has to be the case.

I don't agree, the spread exists for a reason, it is quite literally the minimum cost of doing business for selling liquidity. If selling liquidity were not profitable, no one would do it.

> If the average limit order lost less than the spread and rebate, it would be > profitable to mechanically copy every limit order with an infinitesimally > small limit order placed right behind it.

That doesn't follow. You have no idea how much liquidity is sitting at that price so even posting at the same price wouldn't guarantee execution let alone posting behind it. Limit orders are profitable because they are inherently offering a service that the market is willing to pay for, liquidity, that's it.

> If the average limit order lost more, it would be profitable to copy every > marketable order trading to remove any residual liquidity on the book at > that price.

If the average limit order lost, it'd be called a market order because market orders lose the spread and limit orders make the spread, that's how it works. That is the equilibrium, market orders buy liquidity from limit orders for the price of the spread.

> Of course HFTs limit orders perform better than average or they'd be out of > business (and many do go out of business).

Yes, exactly, and they perform better due to latency putting them first in the queue, that's rather the whole point of HFT, being first in line.

> Speed alone is no longer an edge in most markets.

Incorrect, speed is always an edge for the fastest trader; it's simply no longer easy to compete on speed for most firms. Dodd Frank has devastated HFT with the sub-penny rule making latency even more important and driving most out of business.

> Most are holding risk for more than microseconds or single seconds.

Yes, because they can't be the fastest, they have to start speculating a bit. This is because the hay day of HFT is over; they're being forced into speculation by the inability to compete on price due to the sub-penny rule and the inability to compete on speed due to the massive costs of doing so. But once you start speculating for longer than a few seconds, you're no longer a HFT trader, you're just an algorithmic trader. HFT is market making, if you're speculating on direction and holding for expected moves, you're not a HFT, you're an algo trader.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER57> "Writing computer code, or at the least being conversant in the firm's program of choice, OCaml, is a requisite for all traders."

nytimes... why u so noob?

</comment><comment><USER58> Seriously, are you serious, these are the people who bankrupted an entire generation.

</comment><comment><USER59> Every time I hear about algorithmic trading, I keep thinking back about Enron.</comment>


Chartio develops non-blocking MySQL queries using Tornado - <AUTHOR> http://chart.io/2011/making-mysql-queries-asynchronous-in-tornado/

<comment><USER2> A quick search reveals this post:

[http://jan.kneschke.de/2008/9/9/async-mysql-queries-with- c-a...](http://jan.kneschke.de/2008/9/9/async-mysql-queries-with-c-api/)

with a little more in-depth exploration of the MySQL async API. The problem seems to be that that (undocumented) API does not handle EAGAIN and that there's no way to connect asynchronously (mysql_connnect always blocks).

Compare with PostgreSQL, which has had an async API for a long time:

[http://developer.postgresql.org/pgdocs/postgres/libpq- async....](http://developer.postgresql.org/pgdocs/postgres/libpq-async.html)

and for which I wrote a Twisted library that does fully asynchronous connection building and query execution: <https://github.com/wulczer/txpostgres>.

Since the asynchronous API is exposed in psycopg2, the Python connector library, it should be trivial to hook it up to Tornado as well.

<comment><USER3> It's already done: <https://github.com/FSX/momoko>

</comment></comment><comment><USER4> I swagged this out a couple years ago on a lark for Ruby/EventMachine:

<https://github.com/tqbf/asymy>

I have a much more mature version internally, and my understanding is that you can get gem-ified libraries that do async database stuff for EventMachine today.

I lost interest in this pretty quickly once I added Redis to my stack. Redis is trivial to talk to asynchronously, and by sticking a queue in between your async components and your database layer, it becomes easy to implement the half-async model --- plus you get very smart caching and another layer of indexing "for free".

</comment><comment><USER5> For Twisted, there's <https://github.com/hybridlogic/txMySQL>, or the standard adbapi, which runs queries in a thread pool.

Also, some Twisted documentation was improved after Tornado was released; there's now "Twisted.Web in 60 seconds": [http://twistedmatrix.com/documents/11.0.0/web/howto/web- in-6...](http://twistedmatrix.com/documents/11.0.0/web/howto/web- in-60/index.html)

</comment><comment><USER6> > While this isn't a big deal in nearly all production environments (you want > to know when your database gets slow, so you can optimize queries or add > indexing), it matters a lot to us because a slow query can affect other > customers queries.

One more reason to be leery of single-threaded eventing systems. You'd never run into this issue with a threaded web app, and it would perform just as well provided you kept your datastructures as independent as they are in your current eventing setup.

Comparing an eventing system and a threading system, the eventing system is inherently _less_ shared-nothing. It shares everything the threading system does _and_ the event thread.

<comment><USER4> You appear only to have read the first graf of this post, as the whole point of eventing the database access layer is to eliminate the "slow query blocks" problem.

But that's not why I'm commenting. Rather:

You're able to make that last assertion only by shifting the meaning of the word "shared" and denuding it of all its concurrency implications. Yes, event systems "share" the event loop, in all the glory of the word "shared". However, no two contexts in an evented system ever step on each other for access to a shared resource.

<comment><USER6> > You appear only to have read the first graf of this post

I read a little more than that, but I commented on what was interesting to me. If I'm reading the rest of it right, it's basically a tutorial on making a python extension for two specific mysql API functions. That's fine, but it's not that interesting (to me).

> However, no two contexts in an evented system ever step on each other for > access to a shared resource.

Isn't that exactly what is happening when other requests are blocked by a blocking mysql call? They are stepping on each other for access to the shared event thread resource, which they need concurrent access to. Is this not the case? Please help me understand if I am misreading you.

</comment></comment></comment><comment><USER7> In search of performance we also developed a non-blocking MySQL driver for Python (and gevent). It's used by our real-time web framework, Planet (<http://www.planetframework.com>).

While we could've gotten better performance with more processes, we wanted to stick with one process per core. Given this and a non-blocking driver we naturally got a good performance gain.

amysql, as it's called, is up on GitHub: <https://github.com/esnme/amysql>

</comment><comment><USER8> Just to re-iterate Justin's warnings: while the thought of making everything non-blocking is exciting, this probably should only be used if you're expecting (and can't do anything about it) very slow queries.

<comment><USER4> Really? I've never scaled up a consumer web app this way, but is it really considered sound to insert blocking database calls in async web apps? This sounds wrong to me.

<comment><USER9> It works if the service runs a sufficient number of application instances behind a load balancer. In one of the videos about Tornado, a person from FriendFeed stated that FriendFeed runs multiple instances of the application with blocking calls to MySQL. There were many complaints about FriendFeed, but being slow was not one of them. (See slide 21 here <http://bret.appspot.com/entry/tornado-tech-talk>).

</comment></comment></comment><comment><USER10> This should be called "not-as-blocking" MySQL queries. As they say in the blog post, the calls to send and read still block, but they've made it so you don't call read until there is at least something to read. I imagine this will be an improvement over many setups.

To add to the list of truly non-blocking MySQL libraries, check out Erlang's MySQL driver: <https://github.com/Eonblast/Emysql>

</comment><comment><USER11> This is great. I was going to write my own lib that would use a whole bunch of threads and a non-blocking queue system for executing queries. While I find Tornado somewhat half-baked (it defaults to returning errors in HTML, but was built for handling feeds; its content-encoding header handling is broken, etc), there are lots of applications where non-blocking MySQL would be great.

<comment><USER8> The title is a little misleading. The method here certainly isn't dependent on Tornado, but for explanation/demo purposes it's a common context to explain it in.

<comment><USER11> Yes, I realize. This would work for anything in Python, and possibly other things as well. node.js could ditch thread pool as well.

</comment></comment></comment><comment><USER12> Interesting, Oracle is currently recommending _not_ using async OCI calls in the 11g documentation (altho' the feature is obviously there), instead they recommend you to use blocking calls on separate threads. This means you can get notified when your query completes on the client with no network roundtrip to find out if it's still executing.

<comment><USER13> There's no reason completion notifications can't occur in an async API. It sounds like oracle calls polling "asynchronous", which is not particularly accurate. If the polling operations also involve network round trips to check for query completion, that is a tremendously stupid API.

<comment><USER12> Reason's simple: you have to support in your 11g server clients written in the dim and distant past, so you can't monkey with the protocol too much. You _could_ do this as you suggest by subscribing for DCN events when your transaction completes, but obviously not with SELECTs.

</comment></comment></comment><comment><USER14> With so many database options why would you use MySql?

<comment><USER11> Some reasons:

* It's already in place

* Everyone on your team only knows MySQL

* That's what you know and you are under a deadline

* It's the right tool for the job (as opposed to something very different, from the NoSQL world)

* You are using MySQL Cluster instead of MySQL, but don't have the cash to upgrade to Oracle

<comment><USER15> _"It's the right tool for the job (as opposed to something very different, from the NoSQL world)"_

If I may toot my own horn, I just upgraded the layout of <http://mongodb-is- web-scale.com/> so that you can scroll while the video stays visible.

<comment><USER16> Believe it or not, sometimes mongodb is the perfect tool for the job. It actually does support safe wrriting features, so some of the hype in that video is false. MySQL is always a great piece of the stack but it's not always good.

<comment><USER15> The video also shows its age, MongoDB hasn't stood still in the meantime. I put up a transcript of this movie because I found the mingling of the technical discussion with contemplating working on a horrible farm an interesting concept.</comment></comment></comment></comment></comment>


Ask HN: What day of the week has the most traffic on Hacker News? - <AUTHOR> I'm curious which day (of the week) does HN experience the most traffic/viewership?
If not available, could you wager a guess? I would say, Monday or Friday.
Are there any numbers for this and other HN-related statistics? <comment><USER2> I guess it's Monday, as ppl get back to office :P</comment>


So I took this Wired Magazine reporter to a hackathon and we built this app .... - <AUTHOR> http://www.wired.com/magazine/2012/02/ff_hackathons/all/1

<comment><USER2> "And even as the opportunity to create an app and make money has exploded, the time and budget needed to code that software has fallen closer and closer to zero. Programmers can add core features to apps simply by cutting and pasting a few lines of code. Known as the glue layer, this trick enables them to focus on creating new functionality."

Way to set peoples expectations for app dev, i'll make sure to set all my quotes to $0 and keep the schedule to a couple of days...

<comment><USER3> Exactly my reaction!

</comment></comment><comment><USER4>
         ...software engineers pound away at source code for the 12th straight hour ...
                                                                  +
                                                              ...34 hours to go...
                                                                  =
        ..to present a functioning Android mobile application in 48 hours...

Huh??

        ...I'm leading one of the teams...
        ...We were vying to be one of six "idea generators."...
        ...people with ideas for apps (regardless of experience or qualifications)...

Oh.

</comment><comment><USER5> Yeah Zac! Want to take bets on how long til he starts a startup of his own now that he's seen what its like to rock a hackathon?

<comment><AUTHOR> Hey Danielle!! Maybe down the road here soon, but right now I'm focused on making sure SeatMe is a huge and successful.

(PS.. Still need to talk to you about my plan to integrate Growl and Twilio soon though.)

</comment></comment><comment><USER6> BTW: Right now there is another 48h Android hackathon taking place world wide. I am participating in Berlin at the c-base and hacking an indoor positioning app for Android that will need no knowledge about where the APs are located.

More infos about this hackathon: <http://android-dev-camp-2012.blogspot.com/>

or via #gadcbln (Berlin) or #gadc

</comment><comment><USER7> Anybody know what that Gilt mash up to analyze competitor revenue was? I have a few theories but I'm curious to see what they built.

<comment><USER8> It analyzed a bunch of flash sale sites and showed graphs of when their products sold out.

</comment></comment><comment><USER9> Awesome Zac, sounds like you had some fun!

Ruby not your favorite language? ;)

</comment><comment><USER10> Is there any site that would list hackathon events in SF?

<comment><AUTHOR> I wish there was something like that. I have hooks into a bunch of sources so my plancast has a bunch around the bay area: <http://plancast.com/zbowling></comment></comment>


How to Be a Programmer: A Short, Comprehensive, and Personal Summary (2002) [pdf] - <AUTHOR> https://www.doc.ic.ac.uk/~susan/475/HowToBeAProgrammer.pdf <comment><USER2> Okay, I'm going to be a little bit far out there. A lot of this seems to be grounded in the negative. "How to deal with x" how to "do y when z has failed". Some of it technical and some of it interpersonal. Why have we created this sphere around programming? What makes /this/ profession so different from any other? And my believe is: nothing. It's not hard. It's not better. It won't "eat the world". Heck it probably didn't make that much of a difference in the grand scheme. It's just sitting on your ass and either do as you're told or come up with something better, like 90% of all work. Software is just like any other job, and chances are it is completely stupid like most other desk jobs out there. And if not, the same general skills apply. Stop feeling like it is difficult or even special.

<comment><USER3> No other profession has quite the same capacity to transform itself.

Like blacksmiths, programmers are capable of building tools that not only change the way they work but enable new kinds of work to be done, often by making previously onerous tasks trivial, and giving the individual a much large potential impact.

Unlike blacksmiths, programmers can share their inventions with the entire industry in the blink of an eye. So the rate at which the programmer's toolchain can change is indeed special.

Sure, it can also be glorified plumbing at times, but I think we think programming is special because it is.(Not that that will continue to be the case forever.)

<comment><USER4> You should watch Primitive Technology on youtube: [https://www.youtube.com/channel/UCAL3JXZSzSm8AlZyD3nQdBA](https://www.youtube.com/channel/UCAL3JXZSzSm8AlZyD3nQdBA)

</comment></comment></comment><comment><USER5> "When it comes to actually documenting code itself, as opposed to producing documents that can actually be read by non-programmers, the best programmers I've ever known hold a universal sentiment: write self-explanatory code and don't document code except in the places that you cannot make it clear. There are two good reasons for this. First, _anyone who needs to see code-level documentation will in most cases be able to and prefer to read the code anyway_ ... More importantly however, is that the code and the documentation cannot get our of sync if there is no documentation. The source code can at worst be wrong and confusing. The documentation, if not written perfectly, can lie, and that is a thousand times worse"

This is such baloney. I understand that many programmers want barriers of entry to their chair at the office but rationalizing it is self-deception.

The author is essentially saying to write strcmp() "self-explanatory" rather than the a black box description like: [http://www.cplusplus.com/reference/cstring/strcmp/](http://www.cplusplus.com/reference/cstring/strcmp/)

strcmp() is useless without knowing what the return values mean and reading the code to figure it out is way more time consuming than reading the docs.

<comment><USER6> No, you have misunderstood. This statement refers to documentation in the form of comments in source files. You are referring to the documentation of interfaces (e.g. manual pages in Unix) which is definitely required.

A gnarly function in C will sometimes be impractical to fully 'comment'. English is too verbose and imprecise compared to the code. But, in particular, inserting uninformative comments that explain simple statements (statements that can be clearly understood by reading the line of code) is a known rookie mistake. It can lead to a case of not being able to "see the wood for the trees"—how the code works is still obscure, but every line has a literal translation in the accompanying comment.

Googling "don't comment code" brings up a number of articles and blog posts discussing this issue. One thing people sometimes say is that comments should explain 'why', not 'what', the code does, e.g. an explanation of what the algorithm is doing, where necessary, rather than a description of each programmatic operation. That kind of comment is like a gloss on what the function is doing, or a conversational explanation of the procedure that the code implements.

Again, this is not about documenting interfaces.

<comment><USER5> I'm not too sure the author means about that, even though you interpretation is possible.

"When it comes to actually documenting code itself, as opposed to producing documents that can actually be read by non-programmers, the best programmers I've ever known hold a universal sentiment: write self-explanatory code and don't document code except in the places that you cannot make it clear."

I would say eg. Doxygen style function documentation or that strcmp() doc. I linked falls under documents non-programmers wont read.

Hi is emphasizing writing "little" good documentation over "write crap nobody will read". Is it really a good advice to new programmers about documentation?

Can everyone who have spent their career cursing former coworkers that wrote too much code documentation raise their hands. Congrats you were working in heaven.

I'd say when in doubt, write comments. 'How', 'why' and 'what' ... and 'when' and in what order. But that's a matter of taste of course.

<comment><USER6> The idea that this advice means "don't bother documenting APIs" is a straw man which isn't worth arguing about. The "non-programmers" phrasing is misleading, I agree, but the widely held sentiment that the author was expressing is recognizably the one about comments _in source files_.

When the algorithm itself really needs verbose documentation, or if the code it's intended to be used for instruction (incorporated into a book, for example), there's always literate programming.

The notorious Unix comment mentioned here [https://en.m.wikipedia.org/wiki/Lions%27_Commentary_on_UNIX_...](https://en.m.wikipedia.org/wiki/Lions%27_Commentary_on_UNIX_6th_Edition,_with_Source_Code#"You_are_not_expected_to_understand_this") is actually a good example of a situation where a comment is needed to explain the rationale of the code. At the same time, it was clear to the documenter that the situation was too complicated to be made totally clear by the comment alone (or by reading that piece of code in isolation.) That is the kind of situation in which a comment is necessary, but also inadequate. That is the difficult reality behind this advice: comments don't really work that well when you really want them to.

<comment><USER5> Yeah OK I agree about not wielding strawmans.

Another interesting example on comments is the original implementation of 'cat'. A lot of comments.

[https://minnie.tuhs.org/cgi- bin/utree.pl?file=PDP7-Unix/cmd/...](https://minnie.tuhs.org/cgi- bin/utree.pl?file=PDP7-Unix/cmd/cat.s)

V2 of Unix had no comments in cat. [https://minnie.tuhs.org/cgi- bin/utree.pl?file=V2/cmd/cat.s](https://minnie.tuhs.org/cgi- bin/utree.pl?file=V2/cmd/cat.s)

It's a extreme example, but I know which one i would prefer to debug or modify.

</comment></comment></comment></comment></comment><comment><USER7> I read through some of the Beginner section that I thought seemed inappropriate for beginners and found a number of grammar and punctuation errors. While trying to navigate the file I found that the page numbering shown in the Table of Contents is off by two pages. His advice on "difficult people" sounds like he is talking about a single, specific person but is doing so in a vague way and expects that to function as actionable advice. That seems like a common pattern as I look back on the rest of the section. He talks about what were probably specific events or situations in a very non- specific way and by doing so falls short of providing useful insight. It would be more effective to write about the details than to try to adapt the story to a common experience.

<comment><USER8> And your point is?

<comment><USER9> What is yours? The point of the comment you're replying to is obvious. They provided some specific criticism of the material.

<comment><USER8> I was using a rhetorical device - all I saw was an anonymous complaint about grammar and no real substantive critique with know examples?

</comment></comment></comment></comment><comment><USER10> This appears to be an updated and edited version of this document: [https://github.com/braydie/HowToBeAProgrammer](https://github.com/braydie/HowToBeAProgrammer)

</comment><comment><USER11> I find the "learn to debug" chapter refreshing. I've seen a lot of new hires on web development area who simply don't interactively debug anything, sometimes even don't log, just because "printing" is enough. IMO we need more bootcamps on fundamentals.

</comment><comment><USER12> Lots of previous discussions but not for four years:

[https://hn.algolia.com/?query=How%20to%20Be%20a%20Programmer...](https://hn.algolia.com/?query=How%20to%20Be%20a%20Programmer:%20A%20Short,%20Comprehensive,%20and%20Personal%20points%3E30&sort=byDate&dateRange=all&type=story&storyText=false&prefix=false&page=0)

</comment><comment><USER13> I think comments under this article are more important than linked work itself

</comment><comment><USER14> warning: This is not short.

</comment><comment><USER15> >Short

Proceeds to have 40 pages

<comment><USER16> Most guides are at least 200 pages long. 40 pages is not long in comparison

<comment><USER15> >Most guides are at least 200 pages long.

>Most

Most?

You mean sometimes.

</comment></comment></comment><comment><USER17> I honestly don't even understand the title, write code?

How to become a better programmer makes some kind of sense, though the short answer is similar: write more code.

Which leaves no time to fill the ether with noise, so it's a double win.

How does this end up on the front page in the first place?

<comment><USER18> Saying you'll be a better programmer if you just write code is like saying you'll be a better mathematician if you just write more math.

<comment><USER19> My math skills improved the most when I stopped obsessing over studying the right things, or finding the right textbooks, and just constantly solved problems.

<comment> crooked-v Being a mathematician and solving already-theoretically-understood math problems are two things that often don't have a lot of overlap.

<comment><USER19> Yeah fair.</comment></comment></comment></comment></comment>


Typeform launches form building API called Typeform I/O - <AUTHOR> http://typeform.io/landing/

<comment><USER2> Thanks for posting this on HN! I'm a Developer Evangelist at Typeform and one of two people responsible for building this API. This is a very early version of the API so it will contain bugs and some inconsistency. But if you find anything you don't understand, write here and we'll answer as best as we can.</comment>


Celebrity Deathmatch: Facebook vs. EC2 - <AUTHOR>

I'm wondering what the YC crowd thinks of Amazon's web services (S3 and EC2) as a means to handle Facebook traffic.
The pmarca blog entry about facebook intoned
"the Facebook Platform is primarily for use by either big companies, or venture-backed startups with the funding and capability to handle the slightly insane scale requirements. Individual developers are going to have a very hard time taking advantage of it in useful ways."
Can EC2 and a considered strategy to distribute load across [Amazon's] servers mitigate this?
Are there other tools and services that scale with usage in this way?
Anyone have experience with them? <comment><USER2> EC2 could help avoid the cost of having dedicated servers to handle the potentially massive (and more likely extremely small) traffic from FaceBook. This wouldn't be too hard, just have one dedicated server and a box running a load balancer which would automatically spawn EC2 instances if the average load got too high (and kill them if they got too low).

Although its easy to outline this solution, I do think it would be difficult for a lone individual to develop their application along with maintaining the scaling aspects of it. Worse, I think that the Facebook platform is not a very rewarding venue to pursue for a small company, especially one that would be risking a meaningful portion of their funding on it.

<comment><AUTHOR> But isn't Facebook just a single example of anything viral? More generally, I'm interested in hedging bets against capital costs.

</comment></comment><comment><AUTHOR> My question appears to have sunk without a trace.

I'm surprised because it seems like an issue many folks should be worrying about!

<comment><USER3> The title is horrible... that's probably what did you in.</comment></comment>


How Let's Encrypt doubled the internet's % of secure websites in 4 years - <AUTHOR> https://www.eurekalert.org/pub_releases/2019-11/uom-hle111319.php <comment><USER2> It doubled the percentage of websites serving encrypted connections. That does not at all mean that they are secure websites.</comment>


Ask HN: Need a sleeve for your Kindle 3? Buy one from my mom for $12. - <AUTHOR>

I bought my mom a kindle 3 for Christmas. She loved it so much that she made a little sleeve for it to keep it safe and warm. When we went to meet family and friends for Christmas festivities everyone commented on my mom's kindle sleeve about how great it was and inquired where they could get one too. I asked my mom if she would be interested in selling them online. She was very excited to have her own online business so I built her a quick site. I thought HN would be the nicest place to debut my mother's new entrepreneurial venture.
http://www.rubypouch.com
Thanks HN for being such a wonderful part of my life this year. The hackery/entrepnuery boost you guys give me is something I enjoy so much. <comment><USER2> Not sure if you can correct the following in WordPress.

Whenever I click on the image of the pouch, the title shows "KONICA MINOLTA DIGITAL CAMERA", I am guessing it's the brand of your camera. :P

Thought it would be good to correct it, give it a more descriptive title if possible.

<comment><AUTHOR> Aha! Thanks for pointing that out. I can definitely fix that.

</comment></comment><comment><USER3> I couldn't help but notice that you said that it will keep the kindle "safe and warm". Maybe it's just a misconception from my part but doesn't anything electronic prefer the cold?

<comment><AUTHOR> From what little I understand about electronics, I do believe you are right. I think most electronics do prefer the cold. That is my mother's wording on the site. I'll ask her if she would like to change the "warm" to "snug" or "cozy".

</comment></comment><comment><USER4> Good luck. The official Amazon cases (with or without light) are definitely overpriced, at least in the UK.

<comment><AUTHOR> That's the reason my mother decided to make and sell them. She thought others will be getting kindles as gifts from loved ones and would want to keep them protected without having to spend a lot. In fact she wanted to make the price lower but after seeing the time she put into them I told her that she should price them such that she's properly compensated for her efforts.

I don't have a kindle but I wanted a little pouch for my palm pre and offered to pay her so that I could be her first customer but she wouldn't take my money _smiles_. I'm sure she could make a business making little bags for the various gadgets we have. I was thinking I'd make a page where individuals could make custom size requests for little bags for their gadgets.

Thank you for the comment.

</comment></comment><comment><USER5> Having buy button in every page might give the idea its for sale to a random visitor.

<comment><AUTHOR> I can place the buy button all over but felt that placing it at the bottom of the front page was sufficient. There isn't much on the front page other then the gallery and a paragraph of text.

Would you suggest having it on other places?

<comment><AUTHOR> Taking your suggestion. I placed it in the sidebar and I'm looking for other places to place the buy button that makes sense.

Thank you!

</comment></comment></comment><comment><USER6> The last thing I want for any electronic item I have is for it to be warmer.

<comment><AUTHOR> Point taken. The "warm" is more towards how when you care for someone you want to keep them safe and warm. It's not meant to invoke ventilation blocking imagery.</comment></comment>


Defending Bitcoin, Andreessen Says Mt. Gox Is 'Like MF Global' - <AUTHOR> http://dealbook.nytimes.com/2014/02/25/defending-bitcoin-andreessen-says-mt-gox-is-like-mf-global/ <comment><USER2> it's not much like MF global when you think about it, other than a financial institution failing.

MF global collapsed due to risky behavior by management team. left a lot of assets, and operates in a regulated sphere where principals are known and subject to courts. the scandal of MF global was that the former governor of new jersey, and goldman sachs bigwig, was so arrogant that he though he could prop trade with customer funds...and lost.

Gox collapsed due to incompetent security procedures, if i understand the story correctly. a bank robbery of a sort. outside of the jurisdiction of many of the victims.

we now know that something around 6-7% of the total bitcoins in circulation are now owned through theft. that's a staggering amount. in any normal currency - the heist of a millennium. there's no court system to adjudicate this or try and recover assets for those who lost money.

we know who is responsible for MF global and we know where to go to recover assets. John Corzine might be politically connected, and the folks who lost money might have a hard time overcoming "the system" to get funds, but they know where to go.

And - quick google search - there was $1.5bn in missing funds, which apparently will be recovered: [http://dealbook.nytimes.com/2013/11/05/mf- global-customers-w...](http://dealbook.nytimes.com/2013/11/05/mf-global- customers-will-recover-all-they-lost/)

With Gox? Tough luck I guess. (I lost a token sum, FWIW).

It's enough to make you wonder - if digital currencies are now well understood, flaws analyzed and hacked... and we are starting to know enough to say - digital currency is awesome...

Maybe we should start over with another alt-coin. i'm not particularly thrilled about putting real money or time behind a coin which now has a large chunk of it owned through theft.

<comment><USER3> > we now know that something around 6-7% of the total bitcoins in circulation > are now owned through theft. that's a staggering amount. in any normal > currency - the heist of a millennium. there's no court system to adjudicate > this or try and recover assets for those who lost money.

You say that like it's a problem. Who cares if some people lose their Bitcoins? Nobody is holding Bitcoin involuntarily, after all. Not having a court system to adjudicate this or recover the assets is part of the point of Bitcoin! It's an experiment in anarcholibertarianism, and at least in theory, measures should arise organically in the Bitcoin community to prevent something like this from happening again. The stable price of Bitcoin suggests that the community is optimistic.

<comment><USER4> Comments like this are why I wish "Shit HN Says" still existed on Twitter. Holy crap. The level of pie-eyed idealism is just astounding.

Yeah, it's a good thing this $400M theft just happened! It's a feature, not a bug! Now we can make Bitcoin better! Derp!

<comment><USER5> I think he may _actually_ be coming from a _" You guys made your bed, now lie in it"_ angle, though couched in the language of bitcoin evangelists to disguise the sarcasm.

I would sympathize with that, if not for the fact that buying into bitcoin does _not_ require buying into ideologies.

Or he's sincere, in which case I disagree for the same reason.

<comment><USER3> That makes it sound punitive. I just think it would be a fascinating sociological experiment to let Bitcoin evolve in a way that would allow anarchist ideas about the viability of markets in the absence of centralized power to be empirically tested.

<comment><USER5> I apologize if I misread your intent, you just don't really seem like a _" fuck it, let them have their fun"_ kind of guy.

Anyway, I don't think that declaring bitcoin to be out of touch by the law would be any better an idea than saying that Idaho should do their own anarchist experiment. In both cases you have people who either do not buy into the ideology getting caught in the crossfire _(as well as people who buy into the general idea but do not believe that the prerequisite conditions have been met, or do not believe that the required complementary technologies exist yet.)_

You would at least need to give those people an opportunity to get the hell out of Dodge^W Idaho. Even then though you would be unjustly displacing those people. Imagine if you purchased a house in Idaho, only for the federal government to declare Idaho to be a lawless land set aside for all the 'sovereign citizens' to experiment with. You would not be to pleased.

</comment></comment></comment></comment></comment></comment><comment><USER6> I don't understand why Mt Gox isn't able to pursue the thieves. You can only withdraw funds if you have a verified account. For verified accounts they would have proven the users identity.

It must be possible to work out which accounts have withdrawn more money than deposited.

Obviously the people who did this would have covered their tracks. However that being said, there should be ways that they can account for which user accounts stole coins. Go from there.

<comment><USER7> Are there even thieves? I mean, the whole story makes zero sense.

Supposedly the root of this was multiple withdrawals over the period of months. How is it _remotely_ possible that Gox didn't have a real-time, if not at worst case daily, comparison between their wallet holdings, as reflected in the blockchain, and their outstanding "deposits"? How could that possibly be?

I just cannot buy that story. Saying "they were incompetent" doesn't explain it. The multi-withdrawal thing only makes sense if it was a mass exploit in a short period of time.

<comment><USER8> > How is it remotely possible that Gox didn't have a real-time, if not at > worst case daily, comparison between their wallet holdings, as reflected in > the blockchain, and their outstanding "deposits"? How could that possibly > be?

Not to defend Mt. Gox, but they weren't the only company to be cyber-robbed recently:

[http://www.ibtimes.co.uk/silk-road-2-0-pledges- repay-1-7-mil...](http://www.ibtimes.co.uk/silk-road-2-0-pledges- repay-1-7-million-users-after-bitcoin-theft-1436890)

Apparently, in virtual currency dealings it's possible to be robbed blind and not see it. Pun intended. :)

</comment></comment></comment><comment><USER9> Andreessen: "Mt. Gox has been obviously broken and possibly outright crooked for months."</comment>


The Robin Hood of Science - <AUTHOR> http://bigthink.com/neurobonkers/a-pirate-bay-for-science <comment><USER3> While at university, you just take having free access to journals for granted. When you leave academia the withdrawal hits you. A lot of journals will make an article free _if_ the author pays extra. (For those unfamiliar with how journals work, you submit your work, they get other scientists to review it for them for free, and then they charge you to publish it if the reviewers are all happy enough). The result of this practice is that you don't automatically know if an article will be possible to view or not. You can't just say, "Oh that's in Journal X, there's no point in looking it up because it won't be free". You have to check and be randomly denied!

The effect of this is that you can no longer freely follow the rabbit hole of an interesting chain of references, nor will you read articles outside your bailiwick for fun because they popped up on a blog somewhere. Even if you carefully restrict the journals you're interested in reading, subscriptions will total several thousand per year. Single articles will be priced at $40-50, which is just nuts when you're reading for curiosity. Even if it pertains to your job, it's a chore to jump through the bureaucratic hoops to make your company pay for it. The end result is that, once you leave university, you're cut off from legally obtaining access to a lot of interesting stuff. How does restricting knowledge in this manner serve science or industry?

In physics, arxiv is pretty freakin' awesome... for _recent_ stuff. If you're following a reference chain, you go off the reservation pretty darned quick. I admit, I've been guilty of using some slightly more labor intensive means of obtaining articles without paying for them, but it's a pain. I honestly hope somebody succeeds in making all of history's scientific publications freely and easily available online in a very permanent way. Hell, that research was practically all funded with public money anyways! Why should private companies like Elsevier be collecting money for research Schrödinger conducted on the public dime before anyone alive was born?

Sci-Hub is new to me, but it looks promising. This is something the world needs.

<comment><USER4> It's not a complete solution, but you can walk in to many _public_ university libraries without needing to have a university affiliation and use their computers to read/print articles and/or go find the print copy of many journals.

At the UCSF libraries here in San Francisco you can walk in off the street and browse the stacks and use the photocopiers without showing any ID. You can also use a workstation for one 2h session/day for free to access online journals and databases. You can also pay $100/yr for borrowing privileges (up to 5 items at a time).

[https://www.library.ucsf.edu/services/computing/policies_pub...](https://www.library.ucsf.edu/services/computing/policies_public_computer)

[https://www.library.ucsf.edu/services/borrowing/privileges](https://www.library.ucsf.edu/services/borrowing/privileges)

</comment></comment><comment><USER5> No question scientific publishers are effectively a racket. The problem is that as a researcher you're stuck. If you want tenure, you need a publication in a big journal. There are a few that are open access in some domains, notably biology, but even there Plos Biology et. al. doesn't quite have the cachet of Nature/Science/Cell.

In other fields, it can be even harder to find a good open access solution. In chemistry for example, I can't think of one. You can pay to have ACS or other publishers make something open access, but that's more money out a researcher's pocket that's not going toward better science.

Sure, publishing has costs, but nowhere near what gets charged. Also, much of the labor of reviewing papers is also provided to publishers free of charge, scientists review each other's work for no fee before acceptance in a journal that will charge dearly for it.

We as tax payers pay for this work, the people we pay to do it are caught in a catch-22 situation where they can either hand over their work to someone who will charge an arm and a leg, or be pushed out of science.

<comment><USER6> The researchers deserve some of the blame as well. In physics, at least, and certainly in some other fields, there is a strong tradition of posting preprints to the arXiv or other similar free archives. The journals, including all the top journals, accommodate this---they have no choice.

You mention chemistry. I don't know anything about chemistry journals, but given the precedent set by other fields, I feel that chemists don't have a good excuse for locking their research up behind paywalls.

<comment><USER7> Some chemistry journals do not allow you to publish if you place a copy on a preprint server like arxiv. The key to this battle will be funding agencies.

<comment><USER8> Well... you could still make the copy available _after_ the article was published.

<comment><USER9> If you don't mind breaking your contract with the publisher.

<comment><USER8> What can they do against it?

<comment><USER9> Sue you for damages.

<comment><USER8> You mean, all $0 of them?

<comment><USER10> Stop being obstinate. You know exactly what the parent means, and that publishers can sue authors who distribute the works for which they (the publishers) own copyright.

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER11> I don't want to defend Elsevier, but this article was just egregiously one- sided. Surely an article like this should examine whether this model is sustainable, or what it would really mean if these for-profit publishers disappeared. (Some of them, for instance, employ full-time editors to sift through papers and pick the best ones. How does this sifting happen in the brave new world of universal free access?)

Also they quote "Robin Hood" thus:

"Elsevier, in contrast, operates by racket: If you do not send money, you will not read any papers."

Ummm, that's how most businesses work. If you want a sandwich, you have to pay for it first... Not all businesses are rackets.

"Robin Hood" continues...

"On my website, any person can read as many papers as they want for free, and sending donations is their free will. Why can Elsevier not work like this, I wonder?"

Ummm, because they're a for-profit company, and their officers have a obligation to try to maximize profit for their share-holders?

Again, I don't want to defend Elsevier, but I think a better article would have acknowledged that there's more than one side to this issue.

<comment><USER12> If you don't want to defend Elsevier, then don't. The research was paid for with our tax dollars. The public has a right to read it. The academics who write and review the papers are not paid by Elsevier. The only value Elsevier provides is choosing which papers to accept, and organizing physical printing of the articles. A committee of prestigious academics in a particular field could easily choose a few leading papers to accept each month, and run a web server with some PDFs... and in some fields, that's what's starting to happen.

<comment><USER13> > A committee of prestigious academics in a particular field could easily > choose a few leading papers to accept each month, and run a web server with > some PDFs.

This is a little naive. Running a good journal takes a lot of time and effort (not to mention a little bit of money).

<comment><USER14> Are you familiar with [http://www.nature.com/news/leading-mathematician- launches-ar...](http://www.nature.com/news/leading-mathematician-launches- arxiv-overlay-journal-1.18351)

<comment><USER13> I'm not saying that it can't be done. But I think it is an open question whether we can move to a model where all journals work this way. It inevitably means academics will have to spend time and money on things that they didn't previously have to spend time and money on.

<comment><USER12> Academics already have to keep up with the literature in their field. It is an important part of being a researcher. You don't want to spend your time on something someone has already done, and you want to cite prior work.

In practice, paper selection is done in a bottom-up fashion already. An individual lab usually decides whether a paper is worth publishing. A minor conference decides which of the papers submitted to it are worth featuring. Bigger conferences look at the results of the minor ones. And so on. There is not a guy at Elsevier whose job it is to read the entire internet. Like the rest of their business, it's parasitic off the publicly funded university research system.

<comment><USER13> I know how the system works, I'm an academic myself.

If you really think that no-one employed by journals is actually doing any work, I'm not sure what to say. The reality is that editing and publishing are time consuming, and that if this work is not being doing by paid employees of journals, it's going to have to be done by someone else.

Paper selection is not really the issue. That is largely done by (unpaid) reviewers in any case. It's everything else that goes into running a journal that's potentially going to give rise to problems.

Again, I'm not saying that it's impossible for a group of academics to get together and run a journal entirely by themselves. I think it's less clear whether this model can scale to replacing all current traditional journals. It's very easy to confuse work that's easy with work that's quick. Yes, it's easy to set up a website with a few PDFs on it, and it's easy to forward papers to reviewers, and it's easy to come up with a LaTeX stylesheet for papers, etc. etc. But maintaining all of that stuff and keeping it running smoothly eats up a lot of time, even though there's nothing fundamentally difficult about it.

<comment><USER15> > I think it's less clear whether this model can scale to replacing all > current traditional journals

Yes, it's less clear because we don't have that kind of system yet to see. We need to try it. Perhaps we'll find we do not need all current traditional journals. Perhaps we'll be fine with online archive for all academic publications and a good search engine.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER16> Legacy science publishers are one of the best examples of copyright fail - <USER2> times have changed such that the current rules are no longer aligned with reasonable goals of current society.

</comment><comment><USER17> Isn't this effectively what Aaron Swartz got in trouble for?

<comment><USER18> Except I don't think he actually released any of the documents he was collecting. So what Aaron was doing was far less condemnable. This is just out-right disregard for the copyright law.

And I love it.

</comment></comment><comment> mighty-fine Is it possible for anyone who wants to to just mirror the entire database? How big is it anyway?

<comment><USER19> If each article weighs in at 1mb (probably an overestimate) then 1mil papers weighs about 1Tb, so for their currently reported 47mil papers you are looking worst case at 50Tb. If we assume they have about 50% coverage (which seems reasonable base on this article [0]) then 100Tb total under the 1mb avg. I'm guessing that number is quite high because the true average will be significantly less than 1mb and compression could bring it down even more.

0\. [http://blogs.nature.com/news/2014/05/global-scientific- outpu...](http://blogs.nature.com/news/2014/05/global-scientific-output- doubles-every-nine-years.html)

</comment></comment><comment><USER20> I applaud what sic-hub is doing, but am curious why doesn't Elsevier simply block the IPs that sci-hub is using. I imagine that once a university gets blocked, whoever shared their credentials at that university will get similar treatment to what Aaron Swartz experienced. I for one would never take that kind of risk.

<comment><USER21> Numerous journals offer access to researchers, campuses, etc., through various systems. These are either individually authenticated, or are offered through campus-wide systems.

My understanding is that Sci-Hub works through "donated" access -- either credentials or inside-the-firewall systems -- through which it can request papers. There are other options for feeding papers into the system as well, and once they're inside, they're then available. Given that most disciplines tend to have a few highly-cited papers, and a standard Zipf / Power curve, starting with the most relevant papers and working down will get you a lot of coverage.

You've still got to keep up with new publications, but that's tractable.

Elsevier would have to deny access to entire academic institutions, or highly- placed researchers. Given they're already in a hot seat, that's a risky proposition.

</comment></comment><comment><USER22> Expected LibGen. Was not disappointed.

<comment><USER21> Well, SciHub isn't LibGen, though they're similar in concept. I believe SciHub _used_ to be accessible through LibGen, but the latter has been flattened a bit.

<comment><USER22> True, but there's a strong connection mentioned in the article.

</comment></comment></comment><comment><USER23> The website is:

sci-hub.io

searching for scihub returned results for other organisations.

Anyone use it much? How complete is it? I just tried it and got the nature paper about deepmind's go playing program, which was nice.

</comment><comment><USER24> Although we just had [https://news.ycombinator.com/item?id=11070192](https://news.ycombinator.com/item?id=11070192), this article appears to give more information, so we won't treat it as a dupe.</comment>


Use Cloudkick To Manage Amazon Web Services' EC2 (YC W09) - <AUTHOR> http://www.techcrunchit.com/2009/03/16/y-combinators-cloudkick-offers-simple-cloud-management-system/

<comment><USER2> I am reminded of those who made money in the gold rush by selling picks and shovels to the miners. Those people are the ones who really made the money.

The service looks cool, btw.

<comment><USER3> I can't tell if you think the pick and shovel sellers are Amazon or Cloudkick. They're both selling services towards the greater goal of getting your web idea off the ground.

But I agree, Cloudkick looks great!

<comment><USER2> Good point. Cloudkick is kind of like the guy who sells food to the guy who sells shovels.

</comment></comment></comment><comment><USER4> This kind of thing is certainly useful, and I like to see another player in the game but I don't detect enough differentiation from Scalr, Ylastic or Amazon's own console, or for that matter the Rightscale console.

My 2 cents, I would work on that. The cross-cloudness is quite nice, though.

<comment><USER5> Thanks for the useful feedback. We do offer a much simpler alternative to the AWS Console. No more worrying about key management and security groups, focus on the business. Also we offer graphs that neither the AWS console nor Ylastic offer that give you a better glimpse into your datacenter.

Regardless, we would love to integrate any ideas you have or any infrastructure services woes you've experienced or any general shortcomings of our/other services. Thanks again.

<comment><USER4> Also... what I really want to be using to control my servers are complex rules and behaviors defined and controlled by puppet. If you could make that easy to use on EC2 instances through your console... that would be great!

<http://reductivelabs.com/products/puppet/>

</comment></comment></comment><comment><USER6> Seems pretty cool - but I am a little worried about handing over my amazon keys - those things are literally the keys to the kingdom if you are running your apps with Ec2 and storing data with S3, etc.

</comment><comment><USER7> Congrats! I'm pleasantly surprised to see you've included Slicehost at launch as well. A couple of Qs:

1\. Why is it free? I'm more than happy to pay for something like this (altho I'm wary of relying on a new startup for such mission-critical stuff, see below).

2\. How did you manage to convince an impressive 40 YC startups to rely on your service?

<comment><USER8> Fellow YC companies don't need a lot of convincing for this sort of thing usually, we pretty much just assume fellow companies are making something awesome and use it straight off.

</comment></comment><comment><USER8> Start charging for this. Now. I'm not even using EC2 yet and I want to pay for it.

</comment><comment><USER9> Very cool looking service-

My nervousness comes from a few points- It looks very new so far..

How does this generate it's statistics? Is it running the checks back to a centralized server on their side some place? What's the capacity on that server- If we threw 1200 EC2 nodes to them tomorrow, would it stay up?

What sort of service/support plans do they offer? It's free, but It'd be worthwhile to pay to be able to get someone on the phone 24/7. We pay Amazon's gold support, we'd pay yours if it meant you'd take my calls at 3am.

We normally use Nagios in-house, which is a great application, but has problems adjusting to dynamic load. This looks very decent, but I worry about the uptime.

<comment><USER4> I used Scalr.net for my last startup, and one issue they had twice during the year I paid them $50 a month was... their entire site went down from a DDOS attack. Each time they upgraded their hardware and setup, but overall I think they were running like 98% uptime for the last year. Not quite good enough for the mainstream.

Scalr literally handles dynamic scaling of your application, so it failing is a bit worse than this, though.

<comment><USER10> Did your instances suffer as a result of this downtime? I've been playing around with Scalr, haven't really seen any problems as of late... Would love more input from others who have used / are using the service.

<comment><USER11> I use scalr at the moment(paid service) we have just under 20 instances spread among 6 farms. Been using them for about 3 months now with very few problems. The only major issue I've had is that their DB roles aren't great for large db's(>100gig) though I think they are working on an SBS one.

As it is I use a modified role to allow SBS but have to muck around too much for comfort on start ups since the SBS partitions don't always mount at startup(sometimes up to 10 minutes later) and when they do that the DB init scripts fail which causes scalr to cycle the instance and start the whole process over again. That said I've only had to do that twice since I've used them.

</comment></comment></comment></comment><comment><USER12> How does this compare to Rightscale? Is there an overlap in services?

Either way, great idea... We'll consider moving to it for my startup (NewsCred).

EDIT: Also, very interesting that you don't provide much information or details about your service on the site. Three logos, and three short sentences at the bottom. I was clicking around trying to find out more, but then realized I would have to sign up (I didn't yet). But I can see a lot of people signing up just to take a look.

Was that an intentional strategy? Doesn't it reduce the number of signups or does it increase? Did you A/B test it?

</comment><comment><USER13> We're beta users of Cloudkick too and these guys are great. The most impressive thing is that they constantly ask feedback and act on it too--the hallmark of a company that's going to succeed.

</comment><comment><USER14> This is _great_ took 5 minutes to setup and configure. I am very happy with this. Honestly, I'm not positive that I'd pay for this, with other freely available tools around, but it is very good. Great work.

</comment><comment><USER15> For the devs: When adding a new provider, you should trim the secret key field because I kept copy-pasting from Amazon/AWS and it kept putting in a space character in the beginning of the field.

<comment><USER5> We just deployed that change. Thanks for the feedback!

</comment></comment><comment><USER16> What is the benefit of using this over something like nagios?

<comment><USER5> I'm one of the founders @ cloudkick, and our goal is to eliminate the hassle of setting up and maintaining your own monitoring install.

Nagios is difficult to set up and get right, we abstract away all the details and let you simply click to add more monitoring. Its simple and we worry about the scaling.

<comment><USER4> Now I'm interested. If you could give me really good graphs of different aspects of performance, of an entire hadoop compute cluster at once... perhaps by tying into hadoop... I would love you.

<comment><USER5> We have a bunch of interesting distributed computing problems we are solving in the pipe, so stay tuned. Nothing out of the box yet...

Currently, we have a pretty unique graphing framework. Basically, you can plot any bit of data and send it back to the server, but the limitation is you have to launch the server through us. The feature will be added soon as available to anyone with a cloud server.

<comment><USER4> If I have to launch the instance with you, that pretty much makes using it for compute clusters impossible. One launches those from a script at the shell.

</comment></comment></comment></comment></comment><comment><USER17> Looking forward to checking this out. Have been looking at Rightscale but their fees are steep. Glad to hear Cloudkick could be another option.

<comment><USER4> Your options for EC2 (that I'm aware of) are:

Rightscale, Scalr, Ylastic, Amazon's own console, and now Cloudkick.

But only Rightscale and Cloudkick are cross-cloud.

</comment></comment><comment><USER18> Very cool ... I'm looking to add the panda video platform into some apps and this looks like it might really help.

</comment><comment><USER19> Love this idea of having such a visual layer on top EC2/slicehost. These guys have opened up a whole new market.

</comment><comment><USER20> I think the YC Motto is changing to "Make something others need, and will likely pay for". I have noticed a real nice improvement in the kind of companies being funded. In other words from Good to Great concepts.</comment>


Jessica Alba is Hot, Mathematically Speaking - <AUTHOR> http://www.telegraph.co.uk/news/main.jhtml?xml=/news/2007/08/25/nwiggle125.xml

<comment><USER2> Here's some fun followup on this steaming pile: [http://www.plasticbag.org/archives/2007/09/ben_goldacre_vs_c...](http://www.plasticbag.org/archives/2007/09/ben_goldacre_vs_clari/)

</comment><comment><USER3> 0.69 vs 0.7? If that precision reflects accuracy, how did they get it?

<comment><USER4> I agree... but in all honesty, I don't think I need a mathematically derived formula to tell me that...:D

</comment></comment><comment><USER5> 25/36=0,69444... Looks more like 0.69 than 0.70.

</comment><comment><USER6> "The Jessica Alba sashay beat off competition"?</comment>


Effortless Design - RandomDesigner.js - <AUTHOR> http://nrox.github.io/RandomDesigner.js/

<comment><AUTHOR> This is an experiment with genetic algorithms, and real-time user choices which replaces the fitness function.</comment>