How I made $500k with machine learning and high frequency trading - <AUTHOR> http://jspauld.com/post/35126549635/how-i-made-500k-with-machine-learning-and-hft <comment><USER3> This is just glorified gambling. I am not sure what special insight or advantage he had, other than his own model. Every trader has a model.

It could have easily been called "how i lost 500k with machine learning". Like gambling, it's easy to manipulate statistics to show that you did well in some period of time.

I worked for a large investment bank about 10 years ago, writing trading programs for quant traders who were market makers. The quants called guys like him "retail" investors and they gleefully picked off all those trades. It's how they made all their money.

So, everyone else, beware of making this a case study in how to make lots of money really fast. You are more likely to lose money.

<comment><USER4> It is only gambling in the sense that any business is gambling: Your customers might stop coming tomorrow because the fad wore off, or a competitor provides a better/cheaper/hipper alternative.

(And indeed, living is gambling. It's all just a matter of the risk/reward portfolie).

But jspauld has apparently made $2/trade after fees on 250,000 trading, with a very small standard deviation (I would guess less than $2/trade) - which makes it one of the best businesses one could ever have.

You can't live without gambling - by e.g. going to be a salaried employee for Yahoo rather than Google or that weird newfangled "TheFacebook" thingy back in 2004, was a gamble.

jspauld, statistically speaking, has made less of a gamble there than almost anyone else posting on HN.

> So, everyone else, beware of making this a case study in how to make lots of > money really fast. You are more likely to lose money.

True. But that's true for every single success story posted on HN, reddit, or USAToday.

<comment><USER5> No. Every business has a risk element, but what makes this gambling is that there is no good or service being produced. It's a game of trying to outguess the other players, with one trader's gain being another trader's loss (relative to market returns).

Because there's a commission on trades, and because you pay taxes on net gains but your minimum tax is zero, high frequency trading by its very nature must a loss for most players.

<comment><USER6> As a professional poker player, analyst and journalist, and being fairly well- read on classifications of gambling vs skill game in different jurisdictions, I have not before come across a definition of gambling that was rooted in the idea that "no good or service is being produced."

<comment><USER7> Any zero sum or negative expected return conctract would meet this definition (quantitatively). if it floats your boat (makes you hallucinate etc), of course that might be considered a good or service. so you're right, if I was a lawyer those words would be loose language.

</comment></comment></comment></comment></comment><comment><USER8> I don't have much experience with finance or working experience with machine learning, but I've always wondered how much room there was for a clever amateur to profit in this space, even as it's crowded with much more sophisticated professionals with much more sophisticated algorithms and machines.

I'm thinking back to Garry Kasparov's piece in the NY Book Review a couple years back: [http://www.nybooks.com/articles/archives/2010/feb/11/the- che...](http://www.nybooks.com/articles/archives/2010/feb/11/the-chess-master- and-the-computer/?pagination=false)

He talks about a chess tournament in which it was "anything goes"...competitors could be human, computers, or humans with computers. The expected outcome was that a grandmaster using a Deep Blue-like computer would win, but the winners ended up being a couple of amateurs with three computers:

> _The surprise came at the conclusion of the event. The winner was revealed > to be not a grandmaster with a state-of-the-art PC but a pair of amateur > American chess players using three computers at the same time. Their skill > at manipulating and "coaching" their computers to look very deeply into > positions effectively counteracted the superior chess understanding of their > grandmaster opponents and the greater computational power of other > participants. Weak human + machine + better process was superior to a strong > computer alone and, more remarkably, superior to a strong human + machine + > inferior process._

So in HFT, how much room is there for an amateur to profit over professionals by having a sophisticated process?

<comment><USER9> Good point, I also wonder about the potential to exploit the algorithms used by the "professionals." In other words, if you can come up with a reasonable approximation of what the pros will do, can you use that information to beat them?

<comment><USER10> 'Theoretically', no. Its hard to be optimistic about these two ideas because while the chess example is a good story, its not analogous for many reasons, ranging from disparity in available information to players to a difference of several magnitudes in saturation. Not to mention HFT just isn't chess.

HF traders are just as much hackers as anyone on HN (and there are plenty of HF traders on HN). So 'theoretically', they've already done what is being suggested here. If someone comes along and develops a winning strategy, it really shouldn't be considered as having anything to do with 'professional strategy vs novice strategies'. It would just be about one person either getting really lucky or coming up with something that is genius in its own right.

\--

If there are 'professionals' and then other 'professionals' whose strategy depends on information about how other 'professionals' trade (and there is), you end up with strategies at all valid points in the sample space of possible strategies and counter strategies. Theoretically, there should be no other possible strategies. Inevitably someone will come up with one though, and the 'sample space' will grow. But its extremely unlikely that additional unique strategies are successful just because they 'counter' the strategies in the sample space. But then again, this is real life and these things aren't impossible.

</comment></comment></comment><comment><USER11> There is an air of either incredibility or sheer jealousy in these comments. Nevertheless, I just wanted to tell the OP that he did a great job. Thanks for sharing. I work in the finance industry as a quantitative software developer, and it certainly is not an easy job for one person to do. In fact, I tried (independent of my professional work) doing this myself, and I ended up losing a lot of money. If people are trying to do this, please please be careful. Big companies, like ones I have worked at, have technical and human resources that are vastly more powerful.

<comment><AUTHOR> thanks quant123 ;)

</comment></comment><comment><USER12> Great work, very interesting to me. Counter to what we're constantly told through the media this stuff can be done. Doing it year after year seems to be the elusive part. Intuitively, once you've proven your technique on 1000+ trades it's not luck.

I developed a fully-automated low-frequency stat arb system that I ran in 2007 based on a perhaps even simpler algorithm. It traded various equities equally to the long and short side regardless of market conditions so widespread rally or collapse was irrelevant. I logged about 20-30 trades/day - much slower.

The results, using no leverage, were +90% in a year with a worst drawdown of 2% and a Sharpe ratio of 2. Total trades were 5000+. Month-to-month the results were very consistent until the uptick rule was nixed in July 2007. August 2007 was a record winner for me, but Sept-Dec 2007 fell flat, not losing, but with greatly diminished profits and the same variation and more frequently getting slammed all-long or all-short instead of a mix that was often near-neutral. Also getting fills better than my orders then completely disappeared, as this was the beginning of the HFT middlemen - including your own brokerage. I shut it down at the start of 2008, keeping the profits intact and moving on to other priorities.

I continued to monitor the theoretical results for a couple of years but the conditions didn't return so I eventually cancelled my data feed.

</comment><comment><USER13> Being pedantic, 4000 trades a day isn't HFT. This is stil algo trading, of which HFT is a subset.

I consider HFT to be any strategy where speed itself is the what gives the edge. Colocation is usually a prerequisite, though not sufficient. It's a shame HFT gets all the attention, when it's really a tiny portion of trading activity. Algo-trading in general is 70%+ of market activity in the US.

Also limiting trades isn't really adequate risk management. The tech exists to very accurately model your exposures. This is something I see underdeveloped a lot, and what separates the top trading firms from the rest.

Still I commend you creating a model, working out how to test and execute it automatically and actually trading your own money.

I really think more hackers should be actively managing their money, (in general, not like in the article). We have these amazing liquid markets, all time low spreads/commissions, products like ETFs/derivatives to accurately and cheaply execute a given strategy, and a huge increase in tech to model risk, but personal personal investing is the same as the 60s.

<comment><USER4> In the US, HFT is mostly synonymous with "all out tech war, flooding the order queue so your less-equipped peers get lags". (Nanex publishes analysis on these events, which are not occuring several times a month and keep accelerating).

In Europe, HFT is mostly what OP describes, because they have reasonable control (e.g., you have to have one execution per 10 orders or pay a fine; in US exchanges, you can sometime finds 10,000 orders submitted in 3 seconds, hundreds of thousands per hour, with 10-20 executions).

> Also limiting trades isn't really adequate risk management. The tech exists > to very accurately model your exposures.

That's basically what AIG did with copulas. Unfortunately, the assumptions in these models tend to break during crisis, when correlations go to one. And AIG went bankrupt.

Limiting trades, done correctly (mathematically AND legally) is the ONLY way to do risk management properly. With more assumptions, you can have "more efficient" risk management in terms of leverage (e.g., you can net S&P and RUSSELL exposure by assuming their correlation structure) - but as AIG has shown, that does not mean you are doing a better job of managing your risk.

<comment><USER13> When I say limiting trades, I mean naively saying 'I will have at most x positions outstanding'. Each trade has an associated risk (variance), that interacts in complicated ways in a portfolio, which I'm sure you know.

silly example, 100 small positions could be less risky than 1 large position or, 1 long, and 1 short trade will cancel each other out and create a riskless portfolio (with 0 return).

You need to have a risk budget, account for each trade, and work out the risk for the composite portfolio. Obviously this is not fool proof, but it's a way better approximation of the real world.

<comment><USER4> > interacts in complicated ways in a portfolio, which I'm sure you know.

Yes. And assumptions about this are bound to break at the most inopportune moment, see e.g. AIG, which I already referred to. Read about the "copula model" disaster, as your statement indicates you are unaware of its details. [https://en.wikipedia.org/wiki/David_X._Li#CDOs_and_Gaussian_...](https://en.wikipedia.org/wiki/David_X._Li#CDOs_and_Gaussian_copula)

> 1 long, and 1 short trade will cancel each other out and create a riskless > portfolio (with 0 return)

This is true if and only if the long and short are in the same exchange, AND exchange rules allow netting long vs. short deterministically. Otherwise, you have counterparty risk. E.g., you can be long SPY and short SP contract (in equal underlying), which would theoretically mean your only exposure is interest rate changes (and sometimes not even that!)

However, since this is in different exchanges, it might happen that during a flash crash, your SPY position will be liquidated for insufficient margin at a low price, but then the price bounces back, and you've lost money on a perfectly hedged position.

OP's model (limiting exposure and assuming the worst, if I understand correctly) is not statistically efficient use of margin, but it's way better at actually managing risk than any statistical model.

</comment></comment></comment></comment><comment><USER14> Take a look at the VIX from 2009 until today and you'll understand why you stopped making money.

<comment><USER15> [http://finance.yahoo.com/echarts?s=%5EVIX+Interactive#symbol...](http://finance.yahoo.com/echarts?s=%5EVIX+Interactive#symbol=%5Evix;range=5y;compare=;indicator=volume;charttype=area;crosshair=on;ohlcvalues=0;logscale=off;source=undefined);

very interesting, a spike in sept-dec 2009 that roughly follows his PNL...

<comment><USER16> Without a ton of volatility, any homebrew HFT is going to lose to commissions and spread. Basically, he was trading in one of the few periods where is was possible to make money.

</comment></comment></comment><comment><USER17> I've been considering trying HFT myself for a while. I'm competent with Machine Learning and am a Software Developer by day, so I can program and can sysadmin well enough to get something up and running without any trouble at all.

But, every time I've tried to actually get started, I've always found the amount of research required before being able to begin is just staggering.

It seems like the logical course of single-programmer HFT trading being:

\- Find sample data

\- Build your trading program using sample data

\- When you're happy: connect to live API and set your trading program loose

\- Iterate.

However, the first step and the third step seem like the ones which require the most research.

Is there somewhere which has a straightforward dump of timestamped market data available to download (free or not), in order to actually develop a working program?

Likewise, figuring out what to actually trade with, and which service to use is also pretty taxing.

<comment><USER18> Nothing in HFT is free. People doing this for a living use precision time protocol in a colocated data center to build their own timestamps. However you can get started by buying ITCH, OpenBook, etc... data which has the full market depth feeds for the various exchanges. There are a couple of brokers out there specializing in the space.

If the easy part was building a working model either you got incredibly lucky or the model is wrong.

</comment></comment><comment><USER19> I run a 12 person HFT group in Denver. This is an excellent description of how an individual can bootstrap themselves into success. Great story, and nicely presented.

The one interesting point that he glossed over is what his indicators were. He wrote, "The indicators that were most useful were all relatively simple and were based on recent events in the market I was trading as well as the markets of correlated securities."

Which doesn't really tell you much.

<comment><AUTHOR> Thank you. I'm looking back through my code and there are really a lot of indicators. It would get pretty technical to explain them. They are all explainable it's just that each one corresponds to slightly different market conditions and I just didn't want to get into it.

<comment><USER19> Fair enough. Most people considering trying this probably have a few ideas for indicators.

But for anyone coming to HFT from a coding background instead of a trading background, an explanation of one of your indicators would have been fascinating.

<comment><USER20> Yes, I would find this very interesting. Doesn't matter if the indicator is now defunct.

</comment></comment></comment></comment><comment><USER21> There is a coursera course called "Computational Investing, Part I" that I am taking that aims to build a market trading simulator to test a trading model. It just started so it's not too late to join.

<https://www.coursera.org/course/compinvesting1>

</comment><comment><USER22> A lot of people are stating that this is like gambling - it is - but not in the sense that you think. Firstly he doesn't use his entire bankroll on each trade, secondly he goes long-short consistently over very short periods of time, thirdly he's too tiny to actually move markets, and fourthly he is in and out within a day - where his max var. is 100% on thousands of tiny trades. Think of it like this - he has $100 - he bets $10 of that each day. He can blow that $10 - no problems.

Worst case he runs out of capital over a period of weeks.

He can't blow up in the way that you think - but he can have large drawdowns over a period of weeks.

Markets are eventually consistent scalable systems - and that is why we prefer them over central planning. In the medium term they prices things correctly, cheaply and efficiently (decade+).

In the short term however (sub-decade) - they can't price jack.

Markets are inefficient period - if they weren't, well then P=NP and you could just put your NP-hard problems into a market and get back cheap, quick, accurate results. Oh - wait - protein folding is actually harder than that.

There are 2 major ways to make money in the markets. Value-Growth and statistical arb (often high frequency). The former (Buffett) is highly concentrated bets on the future of business (I'm value - <USER2> TSLA/GOOG/Samsung). The latter is looking for thousands of small diversified statistically significant correlations above 50% (random guesses) and trading costs between securities/price movements over short time intervals (aka statistical ghosts in the data - RenTech/Shaw).

Both work. Both work well. And will continue to do so as long as markets exist.

</comment><comment><USER23> This reminds me of an AMA from a few years ago. Really interesting if you are into stuff like this:

[http://www.reddit.com/r/IAmA/comments/9s9d7/iama_100_automat...](http://www.reddit.com/r/IAmA/comments/9s9d7/iama_100_automated_independent_retail_trader_i/)

<comment><AUTHOR> thanks for posting hadn't seen that

</comment></comment><comment><USER24> Relax with the vitriol. The guy is sharing an interesting personal story, not providing a step-by-step HOWTO or recommending people follow his suit. In fact, the article is really an ad for his startup Courseware.

<comment><AUTHOR> CourseTalk. CourseTalk! <http://coursetalk.org> .. You are right kind of :) But I've made a decision to start reaching out generally so I can attract cool people to work with on whatever projects I may be interested in in the future.

<comment><USER24> By the way, no offense meant by the advertising thing. I enjoyed the article. Although, as a technical person, would've enjoyed more details on the code and algorithms.

</comment></comment></comment><comment><USER25> This article is missing a crucial piece of data: what was the initial investment.. earning 500k with 10k initial investment is genious, with 10M initial investment it's just another year on the stock market.

<comment><USER4> This is a different style of trading than what investors do. He said that he was never more than a few contracts in. A single contract needs $10K day trading margin usually (depends on time frame and specific contract, but it's a reasonable estimate). Therefore, if he was never more than 10 contracts long or short at the same time, the risk was of $100K.

Making $500K on $100K over 6 months is good business.

<comment><AUTHOR> You are correct. I think I had to have $10-30k in my account. I was extremely low risk so they weren't concerned.

</comment></comment></comment><comment><USER26> The charts show he was trading between Jun 2009 and Oct 2010. How much of his gains could be attributed to the market recovery in general? The Dow went from about 7000 to 11000, the Russell from about 600 to 800.

<comment><USER27> Well, in the article he said tat he did not care about direction, he would simple buy when his expected price was up, and sell when down. However, there could have easily been a bias in his model that "preferred" and performed better during upward movements. If so, he got lucky.

<comment><AUTHOR> Longs and shorts were 50/50 and my program showed no preference for up or down days. High volatility and high volume was what it liked.

<comment><USER28> I guess the real question is: what was your alpha in that timeframe?

<comment><AUTHOR> I'm probably showing my ignorance here but what do you mean by alpha? And how is it quantified?

<comment><USER29> Alpha is how much excess return you had over the market (or risk free return(

E.g. if you made 10% when overall market was up 15% for the year, you have negative alpha. [As someone could have bought index and held it through year to generate better return]

If you made 20% when market was up 10%, you have positive alpha.

That is why everyone in the investment community is 'seeking alpha'.

</comment></comment></comment></comment></comment></comment><comment><USER30> Pls do not follow the advice of the OP. I started a hedgefund in 2004 doing HF platform arbitrage and ran it for 5yrs and i can honestly tell you that this is just survivorship bias. This is a very complex field and being off slightly, having a slight bias, a fraction of a point off your execution pricing and a slightly flawed money management system is recipe for disaster.

The biggest issue is the confusion that you can apply machine learning to HF trading. HF trading sub 15min mark is more about playing the deal flow, and only the institutions have an edge on this. This is why goldman had to separate the buy and sell sides in the early 2000's. Above 15mins you are able to find an edge using time series analyses since the market is scaling invariant according to Benoit Mandelbrot and this does not apply to dealflow. Also having access to dealflow allows you to predict volatilty seconds ahead which allows you decrease your risk and increase you reward as well as handle your costs since the volatility will impact your transaction costs even if transaction costs themselves stay the same. There is just so much stuff to cover that a comment will not do justice in explaining what is wrong with this guys logic.

<comment><AUTHOR> There are plenty of shops making tons of money with HFT who do not have deal flow at all - it's got nothing to do with luck.

Survivorship bias would mean I simply got lucky. If you're going to say that you're at least going to need to look at my P&L charts and say how I could possibly achieve that much success with luck alone.

Finally, machine learning has everything to do with my success. There were hundreds of variables in my algorithm that were ALL optimized using ML. If you read the article you would know that I built an accurate model for backtesting that I used to optimize variables as well as confirm that I was going to make money before I even started live trading.

I'm pretty confident that whatever you were doing in 2004 has nothing to do with what I was doing.

<comment><USER11> agreed. jspaulding got it right although I can see nashequilibrium's skepticism as the US futures market is incredibly crowded (read: competitive, no free cookies).

</comment></comment></comment><comment><AUTHOR> If anyone has questions for me happy to answer as best I can.

<comment><USER31> I have one question:

Why doesn't every hacker do this to make extra money? Is it within the grasp of anybody who can program to automate trading like this?

EDIT: Sounds like it's not really for everybody. You have to own or rent a server with access to direct lines to the exchanges, or else your lag will be such that profiting from HFT is impossible. How much do these cost?

<comment><USER4> > You have to own or rent a server with access to direct lines to the > exchanges, or else your lag will be such that profiting from HFT is > impossible.

A profitable predictor is a much, much harder problem.

At a place like Goldman Sachs, a quant with a working predictor gets paid 5 times as much as the IT guy who makes that predictor talk to the market quickly enough.

Because, as your question implies, it is (relatively) easy to do the IT work or hire someone to do it. Not so for the predictors.

<comment><USER32> At a place like Goldman Sachs, you don't need quants or predictors. You call up the CEO of a company you want to post record profits, and you tell them if they don't do absolutely desperate, self-destructive things (screwing employees and customers for immediate gains), you will crash their stock and destroy their entire company. That CEO then has a choice. Be the one who killed the company, or be the one who kept it running for a few more weeks and delivered a record quarter that made Goldman Sachs happy.

Once you have the assets and capacity to actively manipulate the price of any stock at will, the market is a VERY different animal and no longer need to be understood at all. You simply force its hand.

</comment></comment></comment></comment><comment><USER33> Don't do this with your own money. Found a startup building HFT tools, and then raise money for it, and use other people's money to test your tools.

If the tools work, sell them. If they don't, tweak them, try it again, and sell them until they do.

This is risk management.

</comment><comment><USER34> I started the infrastructure for this kind of thing a while ago. It's BSD licensed.

It is a software implementation of the Viable System Model (VSM), a model for autonomous systems developed by Stafford Beer. It provides structure, communications, auditing and alerting for autonomous systems.

Part of it is base code for dealing with stocks and options, treating securities positions as autonomous systems that have the scaffolding for running simulations on themselves. It's in Smalltalk and runs under Squeak and Pharo. It can be found at:

<http://home.rbcarleton.com/rbc/software/smalltalk/VSA/>

I won't try to advertise it as a complete solution, but it might be the start of one.

</comment><comment><USER35> Wait. This is not HFT. There's a huge difference between automated and high frequency trading. What he does is only automated scalping at best (or at the fastest).

Automated trading is more on strategy, while HFT has more to do with volume and speed. With automated trading, you predict price movements. HFT involves being a liquidity provider. You don't use market technical indicators in HFT, you wait for some really huge orders.

HFT firms won't bother him. Those are dealing with an entirely different set of algorithms. He should have contacted brokers instead.

<comment><USER36> quite true. but HFT has become a buzzword like "the cloud" and even many financial industry specialists claim to do HFT but in fact they are just market making. auto-scalping or even hedge funds that day trade with robots are not HFT.

this: [http://books.google.de/books?id=dobO95EBcqsC&redir_esc=y](http://books.google.de/books?id=dobO95EBcqsC&redir_esc=y)

even though its over 10 years old, is real HFT. moving correlations, windowed FFT (of bid and transaction events), microscopic operators, negative first- order autocorrelation of returns.

it is NOT about supplying liquidity or being a market maker. that's just market makers trying to say they are in on the latest trend.

</comment></comment><comment><USER37> Is there a way to do this with Python or Ruby? I could just as well program this in C#, but I have a friend who can code a little, but doesn't really need everything in C# to do what he wants. The value add of offering the simulator, including the taking into account the bid/ask prices and a stochastic model for latency. Combine this with a web based code editor and easy hosting, and I think this would be a viable product.

<comment><USER38> Nope, there is absolutely no way to do this with python or ruby. :)

Seriously though, there are some existing frameworks and products that you could check out. I haven't used this myself, it's just in my bookmarks:

<http://www.rapidquant.com/features>

A C++ library with python bindings: <http://quantlib.org/index.shtml>

And a low level library for data analysis: <http://pandas.pydata.org>

<comment><USER39> If you use their library/SDK/Framework.... how do you actually trade, do you still need a brokerage? Sorry a bit new to this field.

<comment><USER38> Yep, almost any of them have an API these days. Your software would make the list of trades which is uploaded and executed...

Oh, I forgot about another backtesting framework for python. It has a built in IDE as well.

<http://www.quantopian.com>

<comment><USER40> I work at quantopian. We are working on broker integration, so you will be able to trade algos you develop live.

Our backtesting engine, Zipline, is opensource - <https://github.com/quantopian/zipline>

Zipline was unveiled at PyData NYC, and the presentation materials are here: <https://app.quantopian.com/posts/hello-from-pydata>

</comment></comment></comment></comment></comment><comment><USER41> Looking at your first chart there, is there a reason (other than market conditions) you were making significantly more at the end of '09 than mid '10?

<comment><AUTHOR> To be honest I don't know exactly what happened. My theory is that over time more and more market participants started integrating the types of analysis I was doing which rendered my program ineffectual. It's a pretty normal pattern that there is some inefficiency in the market and over time it disappears.

<comment><USER41> Being a machine learning program, how much of it did you tell it to forget?

Were you compounding the data always, or telling it to forget what was going on several months ago? Or somewhere in between?

(I'm pretty unfamiliar with machine learning, apologies if this is obvious or something)

<comment><USER42> This. If the 2009 model worked well, did you try letting your algorithm to 'forget' the 2010 data and see if the model worked better?

</comment></comment></comment></comment><comment><USER43> Could you comment on how your "curve fitting" algorithm worked? Did you end up with an equation for each curve? Im working on something that requires curve fitting and any kind of tip would be helpful.

<comment><AUTHOR> I basically just brute forced it. I came up with a cost function which would measure the difference between a possible curve and each data point. I think you're supposed to do the squared difference but I can't remember if I did that. With a cost function in place it's just a matter of zooming in on variables that minimize the cost function.

</comment></comment><comment><USER44> I am curious as to exactly why the profitability decreased steadily and rather rapidly all the way to ~zero. The article doesn't seem to expound on that unless I missed something.

Is this a result of bots on the other side adapting in some way to what you were doing? I would have thought you would be too small a player for them to notice.

<comment><AUTHOR> I don't think anyone was adapting to what I was doing in particular but rather simply adapting to the opportunities in the market.

<comment><USER44> Thanks. Makes sense... I guess those opportunities tend to get ironed out rather fast then.

</comment></comment></comment><comment><USER45> Very important to understand that making $500k speculatively is not evidence of an 'edge', nor is trading frequency evidence of the absence of luck. From March 2009 through much of 2010, the market was strongly bullish - if his algorithm showed a positive market bias then his returns would primarily be a function of timing (read luck: and there are a million variants on the nature of the bias that could be unwittingly responsible for his returns, despite the frequency of trades).

We cannot even tell if $500k is a good risk adjusted return - we have no information on volatility, nature of the exposure or most importantly how much money he started with?

Not exactly shocked Jim Simons didn't return his email. But completely shocking that he walked away from a successful automated trading strategy... the only thing rarer than a free lunch is a man willing to walk away from one. suspect.

<comment><USER46> 1\. Author graphed his daily returns which should give you a handle on his volatility. He also stated his max drawdown was $2k.

2\. Author has stated elsewhere than he began with $10k in seed capital.

3\. The "bullish" market you cite increased only about %70 in that timeframe while the author's returns were multiples of that number.

4\. Author walked away from a _previously successful_ strategy that no longer produced profits. His montlhy returns went to nearly zero so he stopped.

Edited to fix numbers.

<comment><USER45> 1\. I cannot get even a remote sense for the nature of his risk exposure from looking at his daily returns. 2\. ok. 3\. The point here is that a systematic bias in his algorithm will expose his trading strategy to the good graces of market fortune (luck) regardless of whether he trades a million, billion or once a day. The source of the bias is irrelevant. 4\. did not see where he said that but that very much confirms 'timing' / which in this case I interpret as luck as being at least a contributing factor.

</comment></comment></comment><comment><USER47> _The high point of my trading was October 2009 when I made almost 100k. After this I continued to spend the next four months trying to improve my program despite decreased profit each month. Unfortunately by this point I guess I'd implemented all my best ideas because nothing I tried seemed to help much._

It is of course possible that once you made "real" money with your algorithm it was spotted by the other algorithms which then started working against it. (Aka exploiting it) Having talked with people in that space (hft) I was left with the impression that an insane amount of analysis was done on those trades.

</comment><comment><USER48> Would you be able to open source any of the code behind your trading system? Maybe not the "secret sauce", but it would be interesting to see how you processed the data feeds, modeled the data, entered orders, etc.

<comment><AUTHOR> I certainly could open source it. I may just wait a bit on the off chance that somebody wants to purchase it.

<comment><USER49> that would require some proof that it works today.

<comment><USER4> Not really.

A lot of people in the business would pay e.g. $5,000 for exclusive rights for something that worked this well in 2009 (with proof that it worked in 2009, e.g. verifiable broker statements), and a smaller amount (say, $5,00) for non exclusive rights.

If he claimed it still works but he wants to sell it, it is a completely different game -- because when these things work, they are cash cows.

<comment><AUTHOR> Yes i'm pretty sure it wouldn't work today. By buying the code I realistically mean hiring me to work for them based on what I achieved.

<comment><USER4> The guys you want to work for (2sigma, RenTec, Jane Street, Susquehanna, ...) are unlikely to call you up as a result of this blog post / hacker news exposure/discussion.

If you want to go back to trading, you'll probably have to actively try to get a job -- at the very least, let someone who's still in the business know that you are looking. In my experience in this field, word of mouth and friends-of- friends are infinitely more successful hiring strategies, for both sides.

(Re:releasing the source - I would like to have a look at the strategy, but I would recommend against releasing anything that is even close to being useful, unless you want to spend the next year screening "where can I get a good XTAPI broker" and "I've got XTrader_PRO set up, but I'm getting error 10013, what gives?" emails).

the Nuclear Phynance message board is probably a better place to look for business offers.

</comment></comment></comment></comment></comment></comment><comment><USER50> Trust me, you earned that much because of your luck. Otherwise Andrew Ng would have partnered with another finance professor and they would have been the richest people on earth!! Imagine trading with their expert systems on global markets.

I traded stocks and Forex for years and my experience says, it is not for everyone. What ever indicators,discipline or model you follow it is going to work only if you have the right intuition or luck!

<comment><USER51> I like to trade Forex using mql4, any suggestion? Thanks.

</comment></comment><comment><USER52> You said "Growth stopped" but your P&L shows negative growth approaching zero. Would it be more fair to say that your profitability turned to zero?

<comment><AUTHOR> To be honest I'd given up and moved on to other stuff well before I actually shut down the program.

</comment></comment><comment><USER53> Thanks for the post, it's very inspirational.

Could you explain this part, specifically what do you mean by "bucket"?

"To accomplish this I tracked predicted price moves in 50 buckets that depended on the range that the indicator value fell in. This produced unique predictions for each bucket that I was then able to graph in Excel. As you can see the expected price change increases as the indicator value increases."

</comment><comment><USER53> Thanks for the post, it's very inspirational.

Could you explain this part, specifically what do you mean by "bucket".

"To accomplish this I tracked predicted price moves in 50 buckets that depended on the range that the indicator value fell in. This produced unique predictions for each bucket that I was then able to graph in Excel. As you can see the expected price change increases as the indicator value increases."

</comment><comment><USER54> This is an activity that adds no value to our world.

</comment><comment><USER55> It is pretty clear from his own graph that this stopped working in october '10. which was an eternity ago in terms of financial markets. algorithmic trading has increased manifold since then, so finding another arbitrage opportunity like he did is only going to be more difficult.

</comment><comment><USER56> Cool article but I hope people don't start trying to follow this path. Ask yourself -- why did he stop?

<comment><AUTHOR> yes. and yes.

</comment></comment><comment><USER57> I don't understand trading enough to even understand many of the terms in the article, however I'm curious to one thing: is it possible that a program could be written specifically to exploit yours? And/or is that a potential reason it became unprofitable?

</comment><comment><USER58> Have you ever thought of making a trading system that would buy tons of stock when a flash crash happens? It is going to happen again. If your system is ready and you buy before they shut the market down or roll back orders you could make a hefty profit.

<comment><USER59> The difficulty is in identifying what is a 'flash crash' (i.e. a temporary downward blip in prices caused by computer or human error) and what is a genuine downward price movement.

If the market dives and you quickly get into a big long position, and then it dives some more - what do you do? You can either close out your losing trade and take the loss, or hope that the market comes back up, all the while holding on to the risk of further losses.

Also, there's no guarantee that trades in the middle of a flash crash will remain valid after the crash. The exchange could nullify all trades in a certain period of time, which would completely wipe out your upside potential.

<comment><USER4> > The exchange could nullify all trades in a certain period of time, which > would completely wipe out your upside potential.

This is the most important thing: In every single "flash crash", the exchanges have retroactively canceled trades, in a rather arbitrary manner (e.g., "every trade between 16:30 and 16:38 is null and void"). There is some underlying justification, but it is also arbitrary (e.g., "anything below 3% of the price when the flash crash started", with no specific justification for the 3% number, or a well defined methodology for the time of the crash).

That could easily turn a +$100K profit into a -$500K profit, depending on circumstance.

<comment><USER46> Nitpick: When exchanges have busted trades there is a "at or below $XX.XX" condition as well as the "between XX:XX and YY:YY" condition.

In the Flash Crash as well as the Knight Capital incident "up/down 30% from the Previous Close" was the price collar (anything outside that was busted and anything inside stood).

Of course there is no guarantee that the same criteria will be used the next time around so caveat emptor.

</comment></comment></comment></comment><comment><USER60> The prototypical example of why a tax on financial transactions is urgently needed.

</comment><comment><USER61> Excellent article on this posted here: [http://www.lrb.co.uk/v33/n10/donald- mackenzie/how-to-make-mo...](http://www.lrb.co.uk/v33/n10/donald- mackenzie/how-to-make-money-in-microseconds)

</comment><comment><USER62> Isn't profit meaningless without knowing initial investment? I read the first few paragraphs and got bored. Why not say upfront what the bankroll was to start?

edit: I found what I was looking for in the comments.

<comment><USER63> Just to save someone else from having to look for it like we both did, his initial investment was

      $10k loan from dad. Built it up to 30k trading manually before my automated program went live.

</comment></comment><comment><USER64> @Author ? Can you create an online course and teach us all? Thanks

</comment><comment><USER65> Hopefully not buried too deep, but any books recommended for getting into day- trading, either manual, or algorithmic? Kindle preferred, but definitely not the deciding factor.

</comment><comment><USER66> Might've been what it was a couple of years ago but this post, dated today, is the perfect advertisement for the author's current business.

Well done!

PS: no sarcasm intended, it truly is an excellent advert.

</comment><comment><USER67> I am skeptical for two reasons:

1\. when you have a good system (even one you cannot "improve" further), you don't talk about it. 2\. You don't just stop using it.

There should be more to this story.

<comment><USER68> Can I recommend that you read the article and you will find therein the answers you seek!

</comment></comment><comment><USER69> Out of interest, how much capital did you start off with?

<comment><AUTHOR> $10k loan from dad. Built it up to 30k trading manually before my automated program went live.

<comment><USER70> What were your average transaction costs per trade? 1000 to 4000 trades per day, lets say 2500 on average, translates to about 625K transactions per year. I assume you did not have to pay something like ETrades 3$ commission per future contract, which would result in almost 2M of fees per year.

</comment></comment></comment><comment><USER71> If you're interested in working on this kind of stuff in the San Francisco area, send a resume to Headlands Technologies. email@example.com

</comment><comment><USER72> What contracts did you trade? Looks like you did futures contracts? Did you look into forex at all, or was this strictly equites.

<comment><AUTHOR> I was trading stock market index futures: Russell 2000, DAX, and a bit of Nasdaq. I experimented with other stuff as well.

</comment></comment><comment><USER73> This might be a dumb question but... what system does your algorithm interface with?

Last time I checked, NASDAQ and NYSE don't exactly publish API's

</comment><comment><USER74> Looks like it's written in C# - is that correct?

<comment><USER75> It stated in the article it was C#.

"In 2008 I was "manually" day trading futures using software called T4. I'd been wanting some customized order entry hotkeys, so after discovering T4 had an API, I took on the challenge of learning C# (the programming language required to use the API) and went ahead and built myself some hotkeys."

<comment><USER74> Yes, that's what I was going off of. I was curious about that aspect of it, if it had stayed C# or what.

<comment><AUTHOR> Actually I believe when I switched from T4 to TT it was C++ I was using.

</comment></comment></comment></comment><comment><USER76> Just because you CAN do it doesn't mean you SHOULD. Even if you don't think of it as "gambling", you're still taking in tons of money without providing any tangible benefit to society.

If you want to make money from investing, why not do so in a socially responsible way? Invest in companies that are changing the world for the better. You might not bring home as much money, but at least you'll be able to sleep well at night.

<comment><USER77> You mean, something like Instagram?

<comment><USER78> Not really defending the parent, but believe it or not Instagram provides something that makes a lot of people happy. Winning a bunch of money on the stock market does nothing for anyone but the winner. Not really a fair comparison.

</comment></comment></comment><comment><USER79> "Since I'm no longer running my [half million dollar] program I'm happy to tell all"

This does not ring true.

</comment><comment><USER80> I could do the same with poker. Or I could lose it all. That variance thing is a bitch.

</comment><comment><USER81> Very cool article. Gave a detail explanation of the ENTIRE process. Thanks!

</comment><comment><USER82> I'm glad to see a healthy respect for investment among the hacker community. It's traders like this who commit to nearly a full 10 seconds of ownership that are the backbone of economic growth for this country.

<comment><USER21> I think there is a healthy respect for computer code that can let loose upon the world and make money.

<comment><USER83> Your missing the point that owning part of a company for 10 seconds brings zero social value to the economy. Which I also tend to agree with. Slightly off-topic from the point of the article, which was that he achieved it without backing of a fairly large institution.

<comment><USER21> I didn't miss that point - I just chose not to address it. Lots of things add zero (or negative) social value to the economy including Farmville or the gazillionth Instagram clone. That doesn't mean they're not worth doing.

And HFT or day-trading does bring some value to the world. It enables companies to go to the public markets and raise capital. And investors to sell their shares without having to wait too long for a buyer. But those positives come along with negatives as well.

</comment></comment></comment></comment><comment><USER84> how is it "machine learning"? A few if-else doesn't make it "machine learning", or does it? - @curationary

<comment><AUTHOR> It takes data (last 4 weeks of market update data) and learns how to predict price movements and how to make profitable trades.

</comment></comment><comment><USER85> I'm sceptical that HFT is good for the public. What did you do to ensure your system wouldn't make a flash crash worse?

<comment><USER4> Why is it his business to defend other market participants?

<comment><USER85> I didn't say it was. But everyone should be in the business of not being evil and not exasperating the problems of others. In trading that means not contributing to flash crashes.

I am not saying that this guy's trading did contribute to flash crashes! He may have successfully implemented systems to prevent that. I hope he did, and if so I'm interested to hear how.

</comment></comment></comment><comment><USER86> How much did you start with?

</comment><comment><USER87> Amazing, love your project

</comment><comment><USER88> A rising tide lifts all boats. "Price move predictions" sounds as effective as tea leaf reading.

</comment><comment><USER89> And nothing of value was created.

Just kidding. It was a fascinating article. Thanks for sharing.

</comment><comment><USER90> of course no firm would respond to his noob low yield model.

The level of the coders doing HFT is beyond the comprehension of this guy, added to the team of Mathematicians, Physicists and computer scientists at your average HFT firm, they probably laugh when they read this.

Good try though, it was awesome that eventually he tuned it to profitability, but there's no way in hell they'd buy that amateur software/algorithm.

Kudos though for taking on the task of learning how to code and making money with ML.

<comment><AUTHOR> This is such an odd comment. What motivates you to write such a comment? And what makes my system amateur and noob in your eyes?

</comment></comment><comment><USER91> Basically this is a story about a guy who was smart enough to script up his trading tool (he discovered that there is an API and wrote some code to use it).

He trade other people's money, using other people's (probably employer's) account and resources, I suppose.

His employer have paid all the fees, and, took all the risks - if there is profit - it is mine, if there is a lose - it is theirs.)

The essence of trading is about having a special (insider) position of even being a market maker, who just collecting fees from every trade other people do.)

But this is just my guess.

<comment><USER83> This is not even close to an accurate summary. He never stated that he had any employer backing, and he wasn't collecting market making fees. In fact he was paying brokerage fees which is the exact opposite.

<comment><USER91> He used his own money to test his algos? Come on.

Taking a Machine Learning and Statistics course does not make you a trader. You need something else - access to the system.

This is why my guess is that he was an employee.

<comment><USER92> An employee of what? He states above that he was paying roughly $200/month for a server and $1800/month for the software/data connections to his broker.

<comment><USER91> _Prior to setting up my automated trading program I'd had 2 years experience as a "manual" day trader._ \- mom's and pap's 401k?

</comment></comment></comment></comment></comment><comment> loup-vaillant While this was quite fascinating, I couldn't see this form of trading as anything but a zero-sum game. Some players win, the other lose, like in any other game.

Except finance is supposed to be "serious". In most serious, legitimate activities, extracting money means you provided value somehow. So, what value high frequency trading could possibly provide?

<comment><USER93> A strange position to take, I'm guessing you've absorbed it from the media somehow. His machine was in the market enough to trade 4000 times a day (I would suggest passively, or he would have been eaten by the cost of crossing the spread), so he was basically continuously offering a service to the market - an offer to sell and a bid to buy at the price he thought fair. Do you demand to know what value your local 7-11 provides by selling you milk? The value is that you can buy milk 24 hours a day, so you don't begrudge them the 15c a carton they are making. And they probably charge more than your local supermarket for the convenience too. The dirty thieves.

<comment> loup-vaillant I refine my position here: <http://news.ycombinator.com/item?id=4752742>

I do get the value of trading, investing, and some form of arbitrage. Heck, I don't have to search for the farm to get my milk. That's a service, and it does deserve a reward. What the author did is a bit different:

> _Most of the market volume was other bots that would only execute a trade > with me if they thought they had some statistical edge._

I understood it meant "they would trade with me only if they think _I_ was the sucker". And of course, he would trade with them only if he thinks _they_ were the suckers. It's not providing a service. It's fighting in a zero-sum economy.

Now he did say " _most_ of the market volume". So there's a fraction that's not bots, and probably also a smaller fraction that does not even play the zero sum game, but instead does some positive-sum trading with the mostly- zero-sum players. But this interface boundary seems incredibly _thin_ , compared to the internal zero-sum behemoth. That looks like a highly inefficient use of time, energy, and brains.

And even then, I'm not sure the zero-sum game provides any service to the rest of the world: zero-sum players base their models on the behaviour of each other, not on the actual performance of companies. That would add no new information to the system. At best, that only amplifies the effect of the few that actually predict company performance. And I doubt it does it well.

<comment><USER93> It doesn't really matter _why_ anyone is in the market. The fact is that they are posting prices to the limit order book, and in doing so providing liquidity and accepting risk and providing a service. The only way the OP could get executed passively is by offering a price equal to or better than the market price, yes? So someone got a better price (or more liquidity at the prevailing price) by him being there.

There are really only two actions in a market, providing liquidity (i.e. market-making) or taking liquidity (including arbitrageurs). Stretching the shopkeeper analogy, any time you put a price tag on an item you are taking a risk and exposing yourself. You slap a price on some bananas and then there is a tropical cyclone (it happened in Australia) and the price of bananas doubles. Someone smart swoops in (call them an arbitrageur), grabs your bananas and goes to the till and you legally have to sell them to him even though they're worth double now. You also have inventory risk, if you're holding a lot of bananas out back then there is the chance that they go rotten.

Every limit order (an offer to sell, or bid to buy) has an inherent risk that you should expect to be compensated for (or you wouldn't do it), and enhances the market. The more competition amongst people posting prices the better, it just means more liquidity (you can buy or sell as many bananas as you want) and tighter prices.

The liquidity taker on the other hand is all about exploiting a misprice. Say, Intel release earnings and are down 10%, but the price of the highly correlated ARM hasn't moved yet, if you are the fastest you can take advantage of some poor person who still wants to buy ARM at the same price as before this information was known. Keep in mind here, a price can move without any trades happening (or goods changing hands). The shopkeepers can whip around and change the price tag on their bananas if they are fast enough before anyone buys them.

It is harder to justify the liquidity taker - what marginal advantage is there to having the price of ARM react to that news in 2 milliseconds rather than 20? They force the transmission of information yes, but why is it better to have it happen a bit marginally quicker? On the other hand, how can you have a realtime market without them? It keeps the providers on their toes, and there will always be someone fastest to react.

Incidentally, providing liquidity must have been the chief function of the OP's algo - 4000 trades a day doesn't work for a liquidity taker that has to cross the spread. He isn't getting 2000 mis-price signals a day to swing at, good enough to justify crossing the spread. DAX traded 120k contracts yesterday, he would have been averaging 2-3% of daily flow.

<comment> loup-vaillant Okay, thanks. A few minor points.

I agree that "why" doesn't matter, if you know "what". When you don't however, "why" is a useful predictor.

I understand the value of arbitrage, to some extent. In its current form, it's quite heavy, but I can imagine we're better off allowing it.

We should compensate for value, not for risk. The two are not always correlated. I know trading is risky, but that's not the point. If it doesn't create value, it shouldn't be compensated. And I suspect some forms of trading create little to no value at all.

By itself, closing a deal doesn't mean you provided value. It means the other party _thinks_ you provided value _to them_. When both parties think that, either it's a win-win situation, or someone got tricked. (Delayed bids and offerings complicate this, but it's the same principle.)

If I got it right, the OP did what we could call "short term speculation". If you predict something will rise, buy from a sucker who didn't. If you predict something will fall, sell it to a sucker who didn't. And of course, pray _you_ are not the sucker. Locally, it's totally zero-sum.

Now maybe the whole system facilitates real transactions (with non- speculators) at the boundaries? But even then, for a given volume your reward doesn't seem to be proportional to your facilitation power (which I have no idea how to compute), but to how well you manage to trick your fellow traders. I very much doubt that such twisted incentives can foster a useful, let alone efficient, system.

<comment><USER93> You're still missing it. Every single additional limit order in the market creates _real_ value. If I want to buy 500 shares of company A, and one market participant is offering 250 at $10. Another is offering 250 at $11. It is going to cost me 250 x 10 + 250 x 11 = $5250. Now, the "short term speculator" you are complaining about shows up and puts his tiny little order in, he is offering 1 share at $10. Now the equation if you want to execute is 251 x 10 + 249 x 11 = $5249. He/she just saved you $1. One _real_ dollar. As someone who wants to get a transaction done on a market, more people posting bids and offers cannot _possibly_ hurt you. Only help you. You have the complete freedom to decide to meet their offered price, to cross the spread, to pick up that carton of milk at the 7-11 and walk to the till and make the transaction.

Now maybe that little speculator who sold you the share at $10, he goes and works the bid, he advertises that he wants to buy a share at $9. Someone fills him. It take 2 minutes of trading to work to the front of the queue (easily possible with an equity). During that time, he is exposed to the risk that the stock might spike up in price, a risk you are no longer exposed to since you got your desired trade done $1 cheaper and 2 minutes ago. Then if he succeeds he makes his $1 profit. It doesn't always work - maybe 60% of the time the price ticks up and he just breaks even. He pays exchange fees and clearing on both the in and out trades too.

In theory, you could get involved in all of this with your 500 share purchase. Try to work the bid, get a better price, etc... or if you cross you are effectively paying for a service. Every order posted in the market is a service.

How wide are bid/offer spreads these days? How can that possibly be a bad thing?

<comment> loup-vaillant _(Edit: "limit order" looks like technical jargon. If so, I don't know what it means. Maybe you didn't say what I thought you said)_

In your example, it looked like you wanted your 500 share for something else than selling them. So you're talking about "real transactions (with non- speculators) at the boundaries". So yes, those transactions are a useful service provided to you by the speculators. I'm not denying that.

On the other hand, your wording seems to imply that most transactions are of this type (non-speculator with speculator). As far as I understand the system, they are not. If the OP is to be trusted, the vast majority of transactions are between 2 speculators trying to outsmart each other. Do we at least agree on that narrow point?

My second point is that a such a transaction (between 2 speculators) is zero sum. Locally, because whatever the first speculators won, the other lost. And globally, because wherever the share is, it could still be sold to a non- speculator. Making the transaction between speculators doesn't make it any easier. I'd say it could make it _harder_ , for the non-speculators now have to buy the share faster than the speculators (or pay extra to have traders do it for them). Seriously, where is the value in that?

Now maybe a good speculator tends to provide a good service to non- speculators. However, they are not selected by the quality of the service they provide. They are selected by their ability to rip each other off. How quality of service arises from such a cut-throat competition is mysterious to me. _(Usually, we compete for quality of service directly, so the connection is obvious. Speculating is not the same thing.)_

<comment><USER93> I think we are getting closer to understanding here. You're kind of quibbling over the definition of value in your other post - someone can provide a "valuable" service even if it isn't currently being used (by your 'real transactors'). I might sleep better at night knowing that if I want to liquefy my investment in Apple tomorrow, I can do so without having to worry that nobody will be there to take the other side of the trade, that I'll have to pay an enormous spread or be left holding a position I want to be out of. An international corporation can be more confident of planning their cashflows knowing that the FX market will always be liquid, and will be quoting at most 2 ticks wide in the majors 24 hours a day every day. This is the oil of an economy, the freedom to allocate capital when you want, where you want and pay the smallest possible friction costs to do so.

Many things in our economy can be described as a zero-sum game. Society only needs so much of any given good, and at the retailer level you aren't involved in increasing demand. Two corner stores might get into a price/advertising war, but let's argue that there is still only a fixed amount of product they can sell to a small community. So advertiser's get fat on fees (in our case, exchanges do). So what is the point you ask? Evolution, competition, it can only be good for the consumer. You still haven't disputed that by the way, the consumer wins don't they? How could they possibly have lost? It takes a tin foil hat to think that it isn't better and cheaper to invest now.

" they are not selected by the quality of the service, just ability to rip each other off" - people have a bit of a naive view that there are some sort of magical harlem globetrotter moves you can pull on an order book. The mechanics are dead simple I'm afraid. It isn't chess, not even checkers. You can't even see the other participants, everything is anonymous on the exchanges I've dealt with. Spoofing is illegal, and that is about as clever as it gets. Very little you can do to make your algo better will put it at odds with providing a better quality service.

To prove that, remember the two types of algo. If I write a liquidity provider that _doesn't_ offer a tighter spread than the competition, I will miss out on trade flow and make little or nothing. If I write a liquidity taker that has an opinion that is _wrong_ about the correct value of a security, someone else who is _right_ will smash me. So effectively we are selecting for more liquidity, tighter spreads, and more accurate prices reflecting available information.

<comment> loup-vaillant My definition of value is simple: something is valuable when it manages to raise people's utility functions (right now, that's about as rigorous as I can be). A trade that happens to be closed by a plain old investor is valuable. If it's another speculator, however, the value is zero. Only consequences ultimately matter.

But, if I got you right, the anonymity of it all mean we cannot separate the two… hmm… Then we've got to multiply the potential utility of the trade by the (quite low) probability of it being closed by a non-speculator. Still valuable, but much less. And I'm back wondering to what extent this is worth the (collective) effort. You did change my mind a little, though. I'll need to learn more.

> _You still haven't disputed that by the way, the consumer wins don't they? > How could they possibly have lost?_

The consumer winning does count as creating value. However there are 2 parts in the retail store competition example: the part where they compete on quality, price, diversity… and the part where they pay fat fees to the advertiser. The first part benefits the consumer, the second just add inefficiency in the loop: if both stores could only agree to not use ads, everybody would win.

When Pareto Optimum and Nash Equilibrium are at odds, life sucks.

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER94> Wow, fuck that guy. no really. Should be title "How I stole 500k with machine learning and high frequency trading".

<comment><USER92> Who did he steal it from?</comment></comment>


Why monads have not taken the Common Lisp world by storm (2008) - momo-reina http://marijnhaverbeke.nl/blog/common-lisp-monads.html

<comment><AUTHOR> I [author] must say I'm slightly embarrassed to see this turn up here now. This is not a terribly deep or interesting post. Read [http://marijnhaverbeke.nl/blog/tern.html](http://marijnhaverbeke.nl/blog/tern.html) or [http://marijnhaverbeke.nl/blog/acorn.html](http://marijnhaverbeke.nl/blog/acorn.html) or [http://marijnhaverbeke.nl/blog/browser-input- reading.html](http://marijnhaverbeke.nl/blog/browser-input-reading.html) instead.

<comment><USER2> Don't be. The fundamental observation that monad's usability is tied closely to Haskell's features is important, and still hasn't been taken on board elsewhere. There's always someone trying to do monads in Clojure...

<comment><USER3> Scala has plenty of monads and they work fine.

<comment><USER4> Yes, but doesn't it also have the three requirements mentioned by the article: ML-style function foo, something akin to type classes, and polymorphic return types?

<comment><USER5> In Scala, currying is not all-pervasive. You have to be explicit about it (and it's a definite pain point).

</comment></comment></comment></comment></comment><comment><USER6> > It appears that in the presence of mutable state, a lot of the advantages of > monads become moot.

In the simpler cases where use of monads can be replaced by simple mutable state -- you still lose out on the explicit types of the mutating vs. pure code.

For example, STM is possible because mutating effects are typed, and so can be ruled out of STM transactions.

And in the more complex monads (e.g: transformer stacks), mutable state is just not good enough. You can compose transformers to build things that mutable state simply cannot express, and you'd have to CPS transform your code and avoid mutable state to express those things.

For example, these two monads:

      ListT (ParsecT m)
      ParsecT (ListT m)

Have no corresponding "mutable state" representations.

<comment><USER7> That's a really great point that I rarely see come up. Yes, Haskell makes you "recreate" the RWST monad stack that imitates impure languages... but the same tools also let you build other "language domains" which are fiendishly difficult to implement in non-pure settings (on par with CPS transforming yourself into the mother monad and then working backwards from there).

On Lisp has a chapter devoted to making a leaky macro-based CPS transformer for Common List. ContT adds CPS to any monad stack as quickly as

        newtype ContT r m a = ContT { runContT :: (a -> m r) -> m r }

<comment><USER8> >Yes, Haskell makes you "recreate" the RWST monad stack that imitates impure languages

This is really a point that seems to get lost in most of learning materials on the web. LYAH doesn't even acknowledge transformers. I am pretty new to Haskell but it seems to me that transformer stack decisions are a pretty significant design decision for your application. What has happened to me is almost all the code in my application has wound up inside this stack; the only pure functions are trivial helpers. I'm not aware of what has been written on this topic in terms of guidance or advice. But it seems like it is definitely one of the later-stage humps for a new Haskell developer like me.

<comment><USER7> That's definitely a stage of a Haskell program design. The stack helps to separate out layers of effects, but it can be easy to fix that stack concretely at an early stage and then feel the weight of it later on with overly specialized functions.

The solution is usually to use the mtl-style transformer classes which let you write generalized functions which depend upon capabilities of the transformer stack instead of the concrete stack itself. You end up with a sort of natural dependency injection framework. For instance, here's a function which works on _any_ RW-style monad stack

        augment :: (MonadReader a m, MonadWriter a m) => a -> m ()
        augment a = ask >>= \a' -> tell (a' <> a)

where the Monoid constraint allowing us to use (<>) is coming from the typeclass Prolog involved with the MonadWriter class.

You can then use augment in ANY monad transformer stack which involves Reader and Writer over the same state type.

Note that this defers the extremely important decision about your monad transformer layer _order_ \---the caller of augment decides whether to call it with "ReaderT w (Writer w) a" or "WriterT w (Reader w) a". Reader/Writer commute so it's not a problem, but this changes things in the op's example using "ParsecT (LogicT m)" versus "LogicT (ParsecT m)". If your function depends on a particular ordering of the effects then it must have a less general type.

<comment><USER9> s/Prolog/Monoid/

<comment><USER7> I meant the Prolog-like inference where MonadWriter w m implies Monoid w.

<comment><USER9> I see. That was a bit obscure, though.

<comment><USER7> Ah, sorry, I was accidentally writing in the context of another comment I wrote in this thread where I talked about the typeclass resolution machinery. Without that context, it is pretty out of place.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER10> Incidentally, I am just reading through All About Monads ([http://www.haskell.org/haskellwiki/All_About_Monads](http://www.haskell.org/haskellwiki/All_About_Monads)) and his point 3 ("Allow polymorphism on return types") is what confused me in this example:

      getAny :: (Random a) => State StdGen a
      getAny = do g      <- get
                  (x,g') <- return $ random g
                  put g'
                  return x

I was like.. how on Earth Haskell knows which function "get" should it call? I think this is a point which should be more stressed in the tutorials (I read Learn yourself a Haskell and glanced at couple of others..)

<comment><USER7> It's easy for a Haskell expert to sweep typeclass resolution under the rug, but it's definitely some pretty black magic at first no matter how "simple" its implementation is. The reality is that the compiler works really hard to ensure that it can guess the right "get" and it's pretty possible for it to fail.

Of course, when it fails you know immediately and can remedy it by type annotations, but that can still be challenging.

To answer the question, `get` has its principle type resolved by the type inference engine. In this case, `get`'s most general type is `MonadState s m => m s` and it can resolve what `m` is by unifying it with the type annotation of `getAny`, so we know that we need `get :: MonadState s (State StdGen) => State StdGen s`. From here, the compiler searches through the type class instances in a Prolog-like style to find that `MonadState s (State StdGen)` occurs when `s ~ StdGen`. This resolves more type information and also tells us which definition of `get` is needed—the one that was defined as `instance MonadState s (State s) where`!

The end result is that get ends up with the type `get :: State StdGen StdGen` and is the function defined as `get = State $ \s -> (s, s)`.

<comment><USER10> Thanks. I sometimes wonder, is there a way in Haskell to display a type signature of a function for a particular instance? For example, is there a way to display type of get when used as an instance of State ? I know I can derive it but I would sometimes like to see if I am correct..

<comment><USER11> Some vim and emacs plugins can give you the type of subexpressions like this. Namely, you want the ghc-mod plugin since it provides the hooks into the compiler to query type information.

<comment><USER10> Is there a way to do this in ghci? I know the :t command, but I don't know how to say that I want a specific instance.

</comment></comment></comment></comment></comment><comment><USER12>
      * Try to quickly write the whole thing as a single recursive descent parser. Note the exploding amount of ugliness. Give up.
      * Separate out the tokenizer (novel idea, huh?) to keep parser complexity down. Parser is still a mess. Ugh!
      * Play around with some CL parser frameworks. This helps a bit, but none of the systems I tried produce errors with enough information.
      * Remember the breeze it was to write a parser with the Haskell Parsec library. Mess around with monads for a while, learn a few things, but not how to write elegant parsers in Common Lisp.

I've been going through these exact steps but in JavaScript. Writing languages is fun! And makes me want to kill things!

<comment><USER13> I have no idea what you're working on, but I spent a few weeks this summer writing parsers and trying to figure out how to tackle the ugliness that seems to be inherent in hand-written recursive descent parsers. While I learned a lot, I am not an expert by a long shot. However I did find an interesting technique that is not well covered elsewhere.

I came across this* article, which uses JavaScript as the implementation language, and found it very interesting, in large part because this approach (Top down operator precedence) sort of pulls the precedence hierarchy out of the call graph of a recursive descent parser and into a table, but also because it's an approach that OOP (and JavaScript in particular) is well suited to. I've used it (in combination with traditional recursive descent) in a functional setting (Standard ML) as well, and would use it again, especially for parsing infix expressions (arithmetic expressions, type annotation expressions).

This is a bit of a tangent, but I thought you might be interested given the intersection of parsing and JavaScript. I've been meaning to write this up in a short blog post...

* [http://javascript.crockford.com/tdop/tdop.html](http://javascript.crockford.com/tdop/tdop.html) There's another article on this using Java as the implementation language: [http://journal.stuffwithstuff.com/2011/03/19/pratt-parsers-e...](http://journal.stuffwithstuff.com/2011/03/19/pratt-parsers-expression-parsing-made-easy/)

<comment><USER14> I like the precedence climbing algorithm since you can easily fit it into a normal recursive descent parser and it allows you to easily add new operators just by putting them in a table. See: [http://eli.thegreenplace.net/2012/08/02/parsing- expressions-...](http://eli.thegreenplace.net/2012/08/02/parsing-expressions- by-precedence-climbing/)

</comment></comment></comment><comment><USER15> I came here to make a joke about the real reason why was that no one understands them... but apparently everyone here does. Off to google monads for idiots...

<comment><USER16> This talk by Douglas Crockford is quite informative:

[http://www.youtube.com/watch?v=dkZFtimgAcM](http://www.youtube.com/watch?v=dkZFtimgAcM)

<comment><USER17> As a connoisseur of monad explanations (if not monad understanding) Crockford's talk sounds rushed. It spends a fair amount of time on setup, but when it gets to the meaty bits, it has the feeling of glossing over the details. It could be that all the essential elements are there, but they're covered to quickly to impart any comprehension.

<comment><USER18> Actually, Crockford is completely wrong in his definition. He is confusing them for functors.

</comment></comment></comment></comment><comment><USER19> Nah, I think its just because they result in really nested types which is hard to keep track of without a really good type system. Using a single monad by itself like error or continuation is straightforward in Clojure; there are smart & vocal people who do this. But the super awesome OMFG of monads is that they can be combined (e.g. parser = error + state), and this more or less requires abstracting in the type system to keep track of all the lambdas.

</comment><comment><USER20> >That's a little too blatantly useless to be interesting though. But note how ugly CL's multiple namespaces make liftmaybe and its uses.

Oh yeah, incredibly ugly \s

OP, you may be interested in checking out LiL, the Lisp interface Library.

[https://github.com/fare/lisp-interface-library](https://github.com/fare/lisp- interface-library)

</comment><comment><USER21> There is a point made here about how Haskell's namespace handling makes monads easy, while CL's makes them ugly. I don't know enough about CL's namespaces to understand this. Can someone explain?

<comment><USER20> Functions and other values are stored in separate namespaces so when you want to call a function that is stored in a "regular" variable you must write

(funcall (function variable) arg1 arg2 ... argn)

Or shorthand (funcall #'variable arg1 arg2 ... argn)

Basically, some people find this ugly, I mostly find it to say "HEY!! We're doing this specific thing right here". To me it's more helpful than ugly.</comment></comment>


EU plans first satellite fleet to monitor CO2 in every country - <AUTHOR> https://www.climatechangenews.com/2019/05/27/eu-plans-satellite-fleet-monitor-co2-every-country/ <comment><USER2> > The fleet of three satellites is slated for launch in 2025, in time to > inform the UN's global stocktake of greenhouse gas emissions three years > later

This is a very good project. Having independent monitoring of CO2 and CH4 worldwide using the same method will be useful. It's unfortunate it can't get launched sooner, as data trends are very important (or at least, they're very interesting to me as a layperson observer).

<comment><USER3> Don't really want the EU to become a carbon police.

<comment><USER4> Why not?

<comment><USER5> You won't go on with business as usual if you are being monitored.

I remember something about a town that frequently reported a list of their most polluting local businesses. And the businesses started to become more environmentally friendly just to not appear on this list! There were no fines or other disadvantages, they just appeared on the list.

I'll add a link later if I can find something about it (wish me luck, it seems _ogooglebar_ as Swedes might call it).

<comment><USER6> Businesses becoming environmentally friendly sounds like a positive.

<comment><USER5> I absolutely agree. It was an explanation why some might not even like data being collected.

<comment><USER7> This works in a social setting, a community, a market but it's much less likely to work against countries as a whole. The policy makers respond to local voters (at best) not other countries.

We've named and shamed China for a few years now that it produces 50% of the world's ocean plastic pollution from a single river. Little has changed.

<comment><USER2> If, as someone else said, tariffs were imposed to all countries as a function of greenhouse gas output, it would disincentivise their production, and incentivize cleaning country-wide emissions.

As with most things, there is no silver bullet, but a combination of motivators will create the change we need. Independent data monitoring is the first major step to make it possible.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER8> Coal power generation is still used in Europe, with some countries even increasing usage after Fukushima nuclear event, pulling some nuclear plans. Germany being a classic example of that. [https://www.banktrack.org/news/civil_society_joins_forces_to...](https://www.banktrack.org/news/civil_society_joins_forces_to_push_europe_beyond_coal)

Problem is, countries tend to place such power generating plants near their borders. This in many instances will, with prevailing winds - show lots of CO2 present in a country that did not produce it.

Now the EU has laws and fines for exceeding limits, yet if the wind blows the wrong way - a city/town that by itself sits inside those limits, can tip over those limits. So that city/town ends up with a fine from the EU, not the other source that tipped the balance - which probably has some nice carbon credit deal in-place enabling them to carry on.

Monitoring is good, but satellite monitoring will have its limitations [https://en.wikipedia.org/wiki/Space- based_measurements_of_ca...](https://en.wikipedia.org/wiki/Space- based_measurements_of_carbon_dioxide) when compared to ground based monitoring stations and weather ballons (which can read the layers better).

However - I've not seen a comparison of satellite monitoring in an area which also has extensive ground based monitoring stations. That would be most interesting and insightful. Though I suspect that you could very well see higher levels on the ground that the satellite measures due to water vapour/cleaner air in upper layers defusing the results. Equally you could have clean air at the ground layer and higher pollution in the upper layer and the satellite will show high co2/pollution levels.

Either way, capture at source is needed more and more as many legacy industrial infrastructure pumping it out on the back of carbon credits. ]

<comment><USER9> Nobody in their right mind would even suggest automatically fining countries or cities based on the colour of pixels on a satellite CO₂ map. If a place shows unexpectedly high CO₂ levels, I expect an investigation will happen in order to determine who's emitting it. Weather satellites and ground stations should resolve doubts about wind direction.

WRT. ground stations, satellite monitoring has the nice benefit that you don't have to coordinate with the monitoring territories, or even ask them for permission. The data may be less precise, but it'll be _much_ easier to gather, and much more consistent.

<comment><USER8> >Nobody in their right mind would even suggest automatically fining countries or cities based on the colour of pixels on a satellite CO₂ map.

Didn't say they would, and the point I was making was how a location can go over those limits due to external factors outside their control. Which may well be just above normal in themselves, yet combined with another producing area which in itself would be normal, combined - tip the balance. Sure if you have a clear cut case, I'd expect due process, but when the lines are literally blurred, you get a different picture.

Sure sats do give a nice simple big picture, but currently just 2D snapshots of many layers of 3D space. That may well change over time.

</comment></comment></comment><comment><USER10> >Michael Buckwitz, a physicist at Bremen University and an advisor to the project, said every source of CO2 on earth would be passed over every two to three days.

Given the predictable orbits, what will prevent nations / companies from emitting less on "surveillance hour", and emitting more when out of sight?

<comment><USER11> All the prior emissions will still be downwind and be readily detected. (Source: I supervise people doing this analysis).

</comment></comment><comment><USER12> There is already a terrestrial CO2 measuring network of sensors being built, see the BeaCO2n net [1]. While coverage is sparse so far, it could be greatly enhanced, at cost likely to be cheaper than a constellation of satellites.

I'll be watching these projects as I was contemplating setting up my own amateur atmospheric CO2 measuring station, a not-so-simple task once you consider sensor accuracy and periodic calibration.

[1] [http://beacon.berkeley.edu/about/](http://beacon.berkeley.edu/about/)

<comment><USER11> One CO2 monitoring network that already exists and is mature is Aeronet, which has around 30 sites. The network is used to validate the CO2 satellite data we have now, including NASAs OCO-2.

[https://www.atmos-meas- tech.net/12/169/2019/amt-12-169-2019....](https://www.atmos-meas- tech.net/12/169/2019/amt-12-169-2019.pdf)

<comment><USER11> Argh! Correct network for CO2 validation is actually TCCON. Aeronet is a related set of in situ measurements, in some cases collocated with TCCON sites.

</comment></comment></comment><comment><USER9> This is a brilliant idea and I'm surprised it didn't happen sooner. I thought (but never checked) that weather satellites would have this in their observation spectrum; apparently, they don't. I strongly hope the project will happen, and will happen on schedule.

</comment><comment><USER13> This is a good idea, but I'm a little surprised we can't already do this with in-situ assets.

<comment><USER2> The article shows a graphic of what existing monitoring satellites can do compared to what these satellites will do. For comparison, NASA launched the Orbiting Carbon Observatory 2 [1] in 2014, and it can monitor a swath 10km wide. This announcement is about Sentinel-7 [2], and it will monitor a swath 240km wide. This huge increase in monitoring will significantly improve what it can monitor (eg, montly/annual trends in CO2 production)

edit: It will allow monitoring of all CO2 production sites every two or three days

[1] [https://en.wikipedia.org/wiki/Orbiting_Carbon_Observatory_2](https://en.wikipedia.org/wiki/Orbiting_Carbon_Observatory_2)

[2] [https://en.wikipedia.org/wiki/Copernicus_Programme#Sentinel_...](https://en.wikipedia.org/wiki/Copernicus_Programme#Sentinel_missions)

<comment><USER11> Thanks.

One very important feature that simple in situ monitors don't provide is _total column_ CO2. That's the average concentration from the ground up to the top of atmosphere.

The satellites, and advanced ground measurements, get total column CO2 by observing the behavior of slices of the solar spectrum that are affected by CO2. Basically, the gas absorbs some wavelengths and re-emits in others.

</comment></comment></comment><comment><USER14> When a country has proven higher emissions, there should be consequences. Put a carbon tax on their exports.

That will motivate even the most climate change denying governments to take action.</comment>


Python-like programming language interpreter written in Python - <AUTHOR> https://github.com/akrylysov/abrvalg <comment><USER2> Meta? You want meta? Try byterun [1], a full Python bytecode executor written in Python...

[1] [https://github.com/nedbat/byterun](https://github.com/nedbat/byterun)

</comment><comment><USER3> We had to write a scheme like language interpreter in scheme in grad school. I found it very meta, but interesting.

Northeastern Prof Wand taught the class, a quick google reveals it was 3 book versions ago. Still teaching in Scheme though the book is 2009 (Racket).

[http://www.eopl3.com](http://www.eopl3.com)

And the git source from the book (more useful with the book though...): [https://github.com/mwand/eopl3](https://github.com/mwand/eopl3)

</comment><comment><USER4> Thanks so much for posting this, you've inspired me to try my hand at writing a similar parser. I'll definitely be using your code as reference as I go!

</comment><comment><USER5> Thanks, the lexer implementation helps me a lot for my compiler design course :-)

</comment><comment><USER6> tldr: ?

</comment><comment><USER7> python-like interpreter in python.

 _inception reference_</comment>


Nonlinearities and Success - <AUTHOR> http://mariocaropreso.com/post/62811446044/nonlinearities-and-success <comment><USER2> The traditional name for systems that are deterministic but unpredictable because they can be radically changed by differences in starting conditions that are below the threshold of measurement is "chaos". Or, in human affairs, "luck". We like to draw parallels and deduce conclusions, but a lot of it is down to survivor bias and confirmation bias. We think of some hypothesis of success; soon we see examples everywhere.

It doesn't quite matter: inhuman complexity renders all of them intractable for ordinary comprehension. History is, as Barzun observed, "above all concrete and particular, not general and abstract".

</comment><comment><USER3> All that is so obvious in restrospect that my mind keeps trying to convince me that I always knew it. (Despite evidence to the contrary).

In a competitive environment there can not be a widely known path for success, because the competitors will adapt untill the path changes to something unknown. Thus, if you know about a model, it's probably not valid anymore, and you simply can't study it for long enough to discover the non-linearities.

</comment><comment> dj-wonk I like the article, but the author suggests (or at least creates the impression) that "nonlinearity" causes "path dependence". This is not necessarily true.

Path-independence or -dependence is more about history than the nature of linearity or nonlinearity in variables itself.

From Wikipedia: "A nonholonomic (a path-independent) system in physics and mathematics is a system whose state depends on the path taken to achieve it."

Here is counter-example where non-linearity does not cause path-dependence. Fields that obey an inverse-square law (such as electromagnetism) are called conservative (path-independent) because the work needed to move an object from one point to the other is not dependent on the path.

But, I could be missing something. Let me know!

<comment><AUTHOR> Your reasoning about the electromagnetic field is right. I think the element of confusion here is the definition of nonlinear. Nonlinear in system theory means that the superposition principle is not valid. From Wikipedia: "The superposition principle states that the net response at a given place and time caused by two or more stimuli is the sum of the responses which would have been caused by each stimulus individually". The electromagnetic field in vacuum is linear, because it obeys Maxwell's equations and it is a solution of a set of linear equations. The output is always proportional to the input: you put a bigger charge and you get a bigger field. A linear combination of electromagnetic fields, with constant, real coefficients, is a new field which obeys Maxwell's equations, thus it is still linear and the superposition principle is still valid.

</comment></comment><comment><USER4> That is such a great title.

</comment><comment><USER5> Lotto tickets are the greatest example!</comment>


Shuush - <AUTHOR> http://shuu.sh/

<comment><USER2> Nice idea.

+1 for requesting the bare minimum permissions from my Twitter account. It's not often I see that.

A few commments:

\- 11's Tweets are too small to read.

\- How does it classify users who @reply a lot (not @mention) but write very few "public" Tweets? (As an example, have a look at @atebits[1].)

[1] <https://twitter.com/#!/atebits>

<comment><USER3> I imagine it doesn't distinguish between @reply's and normal messages. I'm making this assumption because it assigned me a Shuush level of 5 even though nearly all of my tweets are @reply's, but other people who tweet "regular" tweets far more often than me were assigned 3's and 4's. I wouldn't be surprised if all it's really doing is taking the two data points of number of tweets and signup date and using that to compute an average tweets-per-day metric.

</comment></comment><comment><USER4> Props for only requesting read access, I wouldn't have used the application otherwise. The idea is good, but there wasn't enough variation among the people I follow for it to be too useful.

</comment><comment><USER5> I like the idea, but the timeline feels a bit weird. Mostly, I wonder why the photos don't change size like the tweets. As the are the ones determining the organization of the timeline (tweets are much too short), I see almost no difference between, say, 4 and 11. It could be better.

Edit: Just realized the effect is better when you skim fast through the timeline.

<comment><USER6> Are there two different sizes used for 11? I have two 11s next to each other, one with font-size 10.135196568048482px, and the other 6.871829374810328px.

</comment></comment><comment><USER7> I agree with some of the other people who posted in reply to this. I think it is an awesome concept because it is always a drag to have to dig through posts from people who put up twenty a day to find the people who put one every once in a while.

I wonder if there isn't another method of sorting you could do. I have few friends so don't know about the large scale application. But possibly sorting them into friends and having their pictures sized differently based on posts. And then click on the picture and you can scroll through their posts. Something along the lines of showing the post from that friend who rarely posts without scrolling down to find it.

</comment><comment><USER8> This is an awesome idea, and like others have said, props for thoughtful permissions management.

Oddly though, I found that the prolific tweeters were the ones I wanted to see, and the infrequent ones were less interesting. I have a core of prolific users (none of whom I have met) that I follow because I like their content, and then an outer constellation of people I follow because I know them. Tweets from the former actually have a better signal-to-noise. Is mine an unusual usage pattern for Twitter?

<comment><USER9> Ditto. e.g. heavier twitter users (at least the ones I follow) have better content overall, or, stuff I want to see. Otherwise I wouldn't follow them :)

The 'light' accounts I follow are mostly personal ones whom I'm interested in. This isn't a bad concept but I would rather it 'highlight' the less-tweeters among us and leave the standard ones alone.

e.g. highlight but don't detract from the busier folks. Probably proves useful for a lot of people but not for me.

</comment></comment><comment><USER10> I'd be more interested in a classifier that learns which tweets I'm likely to be interested in.

</comment><comment><USER11> Like it, helps throw up tweets I'd otherwise have missed. Some tweets are in way to small a font to read. Would also like to see a version where the timeline is not the same as Twitters default time line, but is one based on degrees.

</comment><comment><USER12> Maybe if they made an Android app of it.

Did anyone else see the title and think that it was a new shell?

<comment><USER13> My guess was on shell, but I was pleasantly surprised.

</comment></comment><comment><USER14> The scale should go from 1 to Robert Scoble

</comment><comment><USER15> A nice idea, but it needs a few tweaks. The vertical height doesn't change between a 3 and an 11, so they both occupy the same eye space. It'd be nice if the avatar, text and the entire block were scaled according to the size. Also, there's not much of a size difference (as far as I can tell) between 6 and 11.

</comment><comment><USER16> Hah, Guy Kawasaki's tweets are too small to read. Nice.

</comment><comment><USER17> Interesting, but needs a visual design pass...

</comment><comment><USER18> well, a really nice niche concept!

Scaling up/down might be a bit of problem though. I'm not sure if this logic fits here:

You're shrinking down the tweets of the people who are more frequent,and vice- versa.The first time I used Shuu.sh I didn't notice much(my bad), but after a couple of time turning it on and off, I noticed what is happening.

Will it not make more sense to limit the 'number' of tweets shown based upon the frequency of the tweeter? And the number being decided based upon number or retweets of that tweet or something? Giving user an option of how many tweets from a person of specific frequency would he like to see or determining it on the basis of how many he is following and how many are following those who are being followed by this user.

Uggh.. I just realized, I made it sound more complex that it really is. Umm.. sorry for that ! :D

</comment><comment><USER19> I love the idea, but I would prefer to see the infrequent tweets at the top of the page above the frequent ones instead of the frequent ones being so tiny I can't read them even if I want to.

<comment><USER20> Agree, or highlight the infrequent ones in place.

The people who I follow that don't tweet don't seem to be any more (or less) interesting than anyone else so I'm not sure I'll use it anyway but its a clever idea.

</comment></comment><comment><USER21> Neat. I would like to be able to use the slider (or a similar concept) that you have on the home page so I can just look at the infrequent ones. I want to only look at 8-11 for example.

</comment><comment><USER22> Interesting. Like the concept. Couple of points 1\. The degree 11 tweets are too small to be readable. 2\. Nice to have: color code the tweets based on context.

</comment><comment><USER23> Apparently I follow a bunch of chatterboxes as almost my entire stream is comprised of high volume tweeters. Explains why I don't pay much attention though!

</comment><comment><USER24> Didn't work for my stream: the frequent updaters are the ones who are fun and informative, and the infrequent the ones that say stuff about their days.

</comment><comment><USER25> Cute idea. But my tweets are some of the largest things and I'm not really seeing much else - and mine are pretty frequent, too.

<comment><USER26> I thought so too, but it turns out that I'm following a bunch of blabber mouths. I got ranked a 2!

</comment></comment><comment><USER27> It would be really nice if the title of this submission offered a detail or two other than the domain name.

</comment><comment><USER28> It might just be me, but the people I follow that post infrequently often post nothing interesting.

</comment><comment><USER29> Interesting, just the other day I was wondering if facebook does this right now.

</comment><comment><USER30> this is a really good idea. It makes me wonder about setting my twitter client to follow alternate sources.

</comment><comment><USER31> Looks good, how did it get noticed?

</comment><comment><USER32> great stuff!

</comment><comment><USER33> Interesting concept.

I don't understand why this app needs to access my direct messages.

I'm gonna have to use a fake account.

<comment> mkr-hn "This application will not be able to:

Follow new people.

Update your profile.

Post Tweets for you.

Access your direct messages.

See your Twitter password."</comment></comment>