Using Reinforcement Learning in the Algorithmic Trading Problem - <AUTHOR> https://arxiv.org/abs/2002.11523 <comment><USER3> As someone who has written about this previously [0], worked briefly in HFT before, and read dozens of papers on the subject, I can say with very high confidence that the results are not to be trusted. This paper, just like pretty much any academic paper on the subject, ends with a backtest on historical data, not a real system.

Not only is it (very!) easy to overfit backtests (especially with so little data they are using here), but backtests are nothing like the real world. In the real world there are HFT traders front-running you, latency, jitter, fees, hidden order types, slippage, and a lot of other complexities that don't fit into a short HN post. Whenever you see a paper ending with a backtest you can already assume it's BS.

It's similar to training a robot in an extremely simplified 2D simulation environment without physics or other interactions, and then claiming one has built a real robot. A mistake many people make is believing that trading is all about AI. But in reality, the model often matters less than infrastructure/latency/system/data issues.

In addition to that, people who are actually "good" at trading don't publish papers, they silently make money. Papers are typically published by academics or students who have never built anything profitable but would like to put a paper on their resume. I have yet to see a single good academic paper about trading.

[0] [https://www.tradientblog.com/2019/11/lessons-learned- buildin...](https://www.tradientblog.com/2019/11/lessons-learned-building-an- ml-trading-system-that-turned-5k-into-200k/)

<comment><USER4> > people who are actually "good" at trading don't publish papers, they > silently make money

I've long understood that this was true. It makes intuitive sense.

But are there any cases where it is not true?

Is it possible to "spread the wealth" when it comes to trading, or any money- making endeavor?

Or does it always reduce down to "I win only because you lose"?

<comment><USER5> > Or does it always reduce down to "I win only because you lose"?

It is a zero sum game. Nobody is producing anything, therefore for one to win another must lose.

<comment><USER6> I think thats an overly simplistic view of things. The market is big and many participants trade at different frequencies. Large pension funds need liquidity to move big blocks of stock for their quarterly and monthly rebalances, and the big medium term statistical arbitrage traders provide liquidity for them to do so. HFT players provide liquidity for the stat arb players. The classes of participants with different frequencies actually help one another, while there is competition for alpha within strategies with similar holding periods. Overall the system creates an extremely efficient and liquid system for valuing and exchanging equity - the very system that empowers YCombinator and other Venture investors to make VC investments knowing that their winners will eventually IPO or be bought by public companies.

<comment><USER5> I knew someone would come in with "liquidity".

Many HFT jump out when things get volatile, when liquidity is actually required.

Ultimately HFT is doing nothing of societal value, the race down to zero is never-ending and we are wasting huge amounts of resources on a totally pointless march towards zero. Exchanges should introduce random delays to allow market participants who really want to hedge / buy / sell, then we can shift some of the resources to the real world. The costs required to compete at the lowest latencies are large, and forcing small/medium players out the game, as the investment cost is large, which is also bad.

The system is hugely inefficient. The costs as latencies get lower are ever higher, for an extremely similar end result. The law of diminishing returns.

<comment><USER6> My initial comment was discussing speculative trading in general, but since you mostly brought up some common anti-HFT tropes I might as well address them.

> Many HFT jump out when things get volatile, when liquidity is actually > required.

Do you have a citation on that? If you look the preliminary Q1 results of Virtu Financial [0] (only publicly traded HFT) they seem to be doing more trading than ever in these volatile markets.

> Ultimately HFT is doing nothing of societal value, the race down to zero is > never-ending and we are wasting huge amounts of resources on a totally > pointless march towards zero.

HFT is a mature industry. Latencies have mostly stabilized, and profitability is way down in the last few years. Many firms are merging/consolidating. So in the past few years society is actually spending fewer resources - <USER2> financially and from a human capital standpoint on HFT than it did in the past.

> Exchanges should introduce random delays to allow market participants who > really want to hedge / buy / sell, then we can shift some of the resources > to the real world.

IEX is doing something relatively similar to that for a few years now. They have ~3% of US equities market share. People have the option of trading there but they mostly choose not to.

> The system is hugely inefficient. The costs as latencies get lower are ever > higher, for an extremely similar end result. The law of diminishing returns.

Due to consolidation, costs are actually decreasing. Could it be that the market is... working?

[0] - [https://ir.virtu.com/press-releases/press-release- details/20...](https://ir.virtu.com/press-releases/press-release- details/2020/Virtu-Announces-450-Million-of-Additional-Broker-Dealer- Borrowing-Capacity-and-Preliminary-Quarter-to-Date-Results/default.aspx)

<comment><USER7> > If you look the preliminary Q1 results of Virtu Financial [0] (only publicly > traded HFT) they seem to be doing more trading than ever in these volatile > markets.

Similar story from Flow Traders:

[https://www.flowtraders.com/sites/flow- traders/files/quarter...](https://www.flowtraders.com/sites/flow- traders/files/quarterly_results/2020/Flow%20Traders%20Q120%20Trading%20Update.pdf)

<comment><USER5> Everyone is, volumes are hugely up. The point about liquidity is during the sudden market shifts, not over a quarter!

<comment><USER8> Perhaps you don't follow the news? This was a quarter rich in sudden market shifts.

<comment><USER5> Thanks, I do follow the news. If you read the threads above again you will see that both posters are fully aware of elevated volume, and the distinction was between HFT melting away during short periods of vol and wider "liquidity" from HFT.

So what seemed like a quick drive by wasn't actually correct.

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER9> Former hedge fund and HFT quant trader here. There's a lot of papers to be found claiming some sort of strategy. I don't want to go to cynicism immediately. But we'll get there:

\- Trading isn't just about deciding what to buy and sell, the sexy part that everyone thinks is great. I even had colleagues who thought they were special because they worked closer to the strategies, which meant that certain less glamourous parts were neglected.

\- Less glamourous parts like coding the software to read in the market data and send out orders.

\- Less glamourous parts like schmoozing with brokers to get them to lower your costs.

\- And maintaining infrastructure, which somehow people think should come as part of coding.

Now I'm not saying that RL won't help you. It's just that focusing on the "intelligent" part of the trading system tends to lead to disappointment, as you discover some unknown restrictions on your model that you hadn't thought of. Things like when you find out short selling was prohibited during the period that your model backtest was shorting.

My main red flags when reading papers are:

\- Choosing a dataset from a small market. Basically any market that isn't the US or Western Europe large caps. You'll discover both price impact and high fees quite late in the game.

\- Choosing a very small subset of the market. Smaller n, more noise and overfitting.

\- Short periods. N again.

\- Long intervals between decision making. N again again.

That's not to say there's nothing useful to be read though. You might be inspired by something you come across.

<comment><USER10> Yes, this.

The small subsets and super high Sharpe ratios look suspicious.

Further red flags in this particular case:

\- Completely unclear what kind of data they're using. Are they assuming they can buy and sell one individual contract at the bid price each minute? Or did I miss some crucial information about bid-ask spreads?

\- Abstract mentions a profit, not an information ratio/Sharpe ratio or anything similar.

\- During training they need to tweak the reward function in order to not end up with "buy and hold"? How good is their strategy compared to buy and hold?

\- Plots without proper labels.

</comment></comment><comment><USER11> An additional problem with this is that they use A3C here for trading. A3C is known to not be suitable for adversarial environments (e.g. board games, like Chess).

I wrote a paper that demonstrated that A3C is as exploitable as a uniform random strategy in board games (specifically, some poker variants): [https://arxiv.org/abs/2004.09677](https://arxiv.org/abs/2004.09677)

(Exploitable is a technical term that is defined in the paper; basically, it's "how much can someone who knows everything about your strategy beat you by?")

So I would be very surprised if this survives contact with other traders.

<comment><USER12> > A3C is known to not be suitable for adversarial environments

Interesting! What are the main papers in this area?

Any intuition why this is the case? is it because A2C generally results in brittle policies?

<comment><USER11> It's mostly an issue that A2C isn't designed for adversarial environments. It also doesn't have any notion of hidden information, while other algorithms (eg CFR) explicitly handle this. There's a well-known phenomena of cycling, where agent A will beat agent B which beats agent C which beats agent A; A2C can exhibit this. Think of rock/paper/scissors- AlwaysRock beats AlwaysScissors which beats AlwaysPaper. To avoid this, you typically need to do some sort of averaging.

The alphastar paper and blog post do a good job discussing these issues as they had similar problems. I'd say that's a great starting point (and then following their references).

Blog post:

[https://deepmind.com/blog/article/alphastar-mastering- real-t...](https://deepmind.com/blog/article/alphastar-mastering-real-time- strategy-game-starcraft-ii)

</comment></comment></comment><comment><USER13> Every time such a paper like this comes out, I have to ask myself: if they knew how to make money like this, why would they tell anyone?

<comment><USER14> It's a weird subject for academic study in the first place. Trying do do trading strategies that beat the market is one of the few things that really is a zero-sum game. In the absence of independent scientific interest to optimizing these strategies, what's the point? You might as well study how to optimize strategies for ultimate frisbee or something.

<comment><USER8> It's probably best understood as either job audition, fund-raising, or mathematical entertainment.

<comment><USER14> Sure, I find mathematical finance absolutely fascinating. I just don't think it's worth putting a lot of research energy into. It's usually the boring stuff that matters.

</comment></comment></comment></comment><comment><USER15> It looks like their code is assuming a fixed commission of 20 rubles, which is apparently equivalent to US$0.27. Does anyone know if that's realistic? [https://github.com/evgps/a3c_trading/blob/master/configs.py#...](https://github.com/evgps/a3c_trading/blob/master/configs.py#L24)

Depending on how liquid the market they studied is, code that assumes there is never any slippage may not be very realistic.

There's no comparison to a simple buy-and-hold strategy, which may be less interesting from a computer science perspective but is a good way to avoid spending lots of money on transaction costs.

(I once posted my own algorithmic trading project to HN that had very flawed, naive assumptions about what trades could be executed.)

<comment><USER16> IIRC moex is perticularly expensive to trade on, and costs can be non linear -- but something like 1bps of commms is a reasonable approximation as an upper bound. In the paper they claim a cost of 2.5RUB per transaction, not the 20 in the config file.

Edit*: It looks like the comms are indeed around 10RUB per side (which is approx 1bps). [https://www.moex.com/en/contract.aspx](https://www.moex.com/en/contract.aspx)

If the model is trading single lots then this is a reasonable cost assumption, otherwise it isn't. The paper using 2.5RUB as costs is unreasonable.

</comment></comment><comment><USER17> This is just fitting on noise. The vast majority of movements are random and no more predictable than a coin flip. Before training, your job is to extract that extremely weak signal, _then_ train.

Try generating a time series in Excel with Brownian noise, watch as it is indistinguishable from price charts.

<comment><USER18> What if someone did find a pattern in Brownian noise...

</comment></comment><comment><USER19> Anyone interested in delving into reinforcement learning should check out our book [https://www.manning.com/books/deep-reinforcement-learning- in...](https://www.manning.com/books/deep-reinforcement-learning-in-action) \- we cover actor critic (used in this paper) to multi-agent to relational/attention models

<comment><USER20> Thanks, looks very useful.

Since you seem to be industry practitioners: I moved away from RL 10 yrs back disillusioned with lack of real-world applicability. Has that changed significantly? The only major name I've heard of is Vowpal Wabbit. Maybe there are more applications being done in stealth. Any insight? Thanks

<comment><USER21> You might be interested in the recently-launched Covariant ([https://covariant.ai/](https://covariant.ai/)), they apparently actually have systems in production. Pieter Abbeel is one of the founders and they have some pretty "heavy" investors, like Jeff Dean, Geoffrey Hinton, and Yann LeCun.

<comment><USER20> Thanks! OpenAI has made some progress on this problem too: [https://news.ycombinator.com/item?id=21259765](https://news.ycombinator.com/item?id=21259765)

Looks like these two places are on the cusp of a major breakthrough in RL/robotics!

</comment></comment></comment></comment><comment><USER22> i've been trading crypto in large volumes at high frequencies for quite some time now. my models were plain as yogurt feed-forward neural nets. i would engineer some dumb features, sample the at random data, assign the labels (that translate into trading decisions), and train the model. then push to prod, sit back, and relax while the balance grows like a mushroom cloud. just kidding, before that i would grow gray hair while backtesting, debugging issues, etc.

one of the hard problems was labeling the data. knowing that the price is going up 10 bps one minute from now, should i buy? maybe. but what if it's going to crash 100 bps right after this? probably should sell instead.

reinforcement learning promises to eliminate the need to assign labels in the training data. the agent will try a bunch of different variants at random and eventually will choose the most optimal one knowing the state of the working, i.e. the state of the markets. at training time i only need to feed it the features data. another benefit is that backtesting and model training is sort of fused into a single process. rl model is optimizing pnl, and not the label classification score (as in the nn model). with proper train-test-validation split, the most performant rl model can go straight into production (helping me to keep some of my hair brown)

while all the bits and pieces seem straightforward i never managed to tune rl model to work better in the backtest compared to the good old old nn models. maybe i have never been closer to the gold vein, but for now, i abandoned my efforts to build a performant rl agent if favor of nn models.

amen.

</comment><comment><USER23> Many people are criticising their backtest but I don't understand why. Their test data is sequential to their training data, considers time, and doesn't overlap. They can't overfit to their test data. In any other area of ML this would be an acceptable scheme, why is this unacceptable here?

<comment><USER3> What people are complaining about is not the overfitting, but the unrealistic assumptions in the backtest. In the real world there is slippage, latencies/jitter, special market open regimes, hidden orders, market impact, front-running, variable fees, and all kinds of other complexities. Their transaction costs are apparently also an unreasonable assumption. Sophisticated simulators used in professional trading firms can account for such things to some extent, but most academic papers conveniently ignore these complexities and just assume they can trade at whatever price the data tells them. It's completely unrealistic.

To answer your original question about overfitting, they can still overfit to test data by running a lot of experiments with different hyperparameters, architectures and parts of the data, and only report what has worked. There are also more complex ways that test data can leak into training data (see the book Advances in financial ML for a good overview). You can already see this is likely the case just from the variance in their results and trades. They also don't compare to baselines. It's not unlikely that the results are just random and they fail to report those experiments that didn't work. Of course, you cannot prove this without having an exact log of all things they ever did to the data. But again, that's not the main issue here.

<comment><USER23> > They also don't compare to baselines

This is certainly worth criticising

> they can still overfit to test data by running a lot of experiments with > different hyperparameters, architectures and parts of the data, and only > report what has worked.

but this is a different accusation from accidentally overfitting or leaking, i.e. it would mean that they're dishonest and cherrypicked their data in such a way that it hides overfitting and leakage. This criticism can be levelled at every ML paper, but in this case they detail their architecture, provide the code, and provide a Jupyter notebook to let people try it themselves.

> just assume they can trade at whatever price the data tells them. It's > completely unrealistic.

I think that this is a fair assumption for highly liquid markets and relatively small trades, and if it's a fair assumption then all of your criticisms (slippage etc) don't apply to the extent that they'll break the approach. Also, if the approach works then trade size (fees aside) and being frontrun also wont apply because presumably large HFT firms can use it.

Overall I think your criticisms are valid, but imo they don't invalidate a promising approach, they're just the next thing to test.

</comment></comment></comment><comment><USER24> Problem is, there is competition, so any "feature" your ML has discovered, will fade away as other MLs discover it too. So it has a tendency to become a completely featureless stochastic system. Temporary features can exist though.

</comment><comment><USER25> All profitable automated trading strategies that I'm aware of target a specific inefficiency in the market. What is the inefficiency here? If you can't articulate the inefficiency, it's probably best not to employ the strategy.

<comment><USER26> Interesting, could you describe one inefficiency that was exploited in the past? I could imagine buying/selling due to spreads between exchanges but is there another not as obvious example?

<comment><USER27> I can describe one inefficiency from sports gambling. There is a famous NBA Gambler named Haralabos Voulgaris. He realized that the points total prediction for a game, let's say 100 points scored for Team A, was merely sliced in half to represent the half-time score. However, the pace of the first half is markedly different from the pace of the second half, thus points are scored at an uneven clip. He exploited that inefficiency for a while to great success.

Like sports gambling, a lot of the financial products we trade are obviously built by humans using rules, and arbitraging the intrinsic rules and regulations around said products. Think about Forex trading where you convert currency into currency. One of the key strategies is to find and identify brief negative cycles, for example, in the hope that converting US Dollars to Euros to Yen back to US Dollars leaves you with more dollars than you started out with.

</comment></comment></comment><comment><USER28> So many papers by "smart" people predicting the past

<comment><USER23> How do you propose that anyone evaluates their algorithm on the future?

<comment><USER28> start a fund with your ML approach - evaluate how much money you've made after 1m,6m,1y,2y,10y etc..

</comment></comment></comment><comment><USER29> I'd like to know what Rentec does. Or whether Rentec even does HFT?

</comment><comment><USER30> Human greediness is endless. Using AI for stock trading is just unethical and waste of resources. We are sick.

<comment><USER31> AI has been used in stock trading for decades now. Also, why does it even matter? Why is it unethical to use AI for trading to begin with?</comment></comment>


A lack of education could be just as dangerous as smoking, study says - <AUTHOR> http://www.washingtonpost.com/news/to-your-health/wp/2015/07/08/a-lack-of-education-could-be-just-as-dangerous-as-smoking-study-says/ <comment><USER2> Correlation is not causation.

From the article "The death counts are an estimate of education's impact on mortality, and do not indicate direct causality."</comment>


Y Combinator for non-techies - <AUTHOR>

What do you think of creating a YCombinator-like venture to help startups that are not founded by techies or are not in the tech-related field? <comment><USER2> It's a great idea, but you'll benefit from focusing on a particular industry (so that the startups benefit from network effects & relevant mentoring/advice etc.). Also, one of the reasons YC works so well is the relatively low fixed overheads & scaling effects of success in software. If you can find a non-tech niche that fits then fantastic - good luck!

</comment><comment><USER3> We occasionally do fund startups led by non-techies. They should be doing something at least related to technology though, because it's hard to really be a startup without that.

<comment><USER4> Can you elaborate on Y Combinator's definition (or scale) of techie -> non- techie?

Is someone with a knowledge and understanding of the tech landscape and years of experience in the industry considered a techie...even if they're not hackers? Or does Hacker=Techie?

<comment><USER3> I just meant non-hackers.

</comment></comment></comment><comment><USER5> One idea for a non-tech field where it might work: film!

All the prerequisites are there:

1\. Loads of young people with good ideas but needing funding, advice and connections

2\. $20,000 is enough to make a worthwhile full-length film nowadays (have you seen Primer? It cost $7,000.)

3\. A successful yet cheap film can easily make back many, many times its production cost.

4\. A single well-connected individual (the pg of film) could provide enough connections to give the films the maximum chance of being shown at festivals etc and eventually picked up for distribution. Alternatively, even the ones that weren't picked up for distribution could be marketed on DVD, and the fact that they were made as part of the "YC-of-film" program would attract at least some interest for them.

Foreseeable problems: the existing studios and unions might have a problem with it. I'm not sure exactly what they would do about it, though.

Conclusion: If I were a bored, well-connected Hollywood guy with money to throw around I'd be totally doing this right now.

</comment><comment><USER6> I think the reason tech startups (and web-tech startups in particular) work great with the incubator/YCombinator model is that $10,000 and 3 months gets you somewhere in terms of a beta or an alpha, or a design or something that can get you Angel Investors or SeriesA VCs. Many non-techie businesses don't really have that "alpha" stage until there's a lot more money involved, making 3 months and $10,000 a poor trade for equity from the founder's perspective.

Additionally, tech startups tend to have a serious valuation ramp-up and multiple investment rounds that many other industries don't. So getting in on the ground floor with $10,000 is a worthwhile investment. For a YC-like program to be successful, it has to be in a field with a similar structure.

</comment><comment><USER7> I say we make programming languages where you don't have to program.

Seriously, though, if they aren't in the tech world...they are just waiting to be overtaken by us....agriculture? could use better tech.... medicine? could use better tech.... massage therapy? we could add a muscle scanner...

Maybe a place for non-webapp-oriented tech businesses, like biotech and such, would make for a good HN spin-off...or maybe we just need more of those here...

.: no sleep until the machines and we are one ! :.

</comment><comment><USER8> While the founder might not be a techie, if a new business in a traditional field could get a leg up on the establishment by the adoption of emerging, appropriate technology, that might be worth considering.

Woz was the techie. Steve Jobs wasn't. Yet Jobs, while only a co-founder, turned out to be the, uhm, core of the long-term Apple success.

(Afterthought: Is it even possible to start a business these days and hope to compete without substantially embracing _some_ emerging technology?)

</comment><comment><USER9> That's the idea behind 14ninetytwo (<http://www.14ninetytwo.com/>) here in Columbus, OH.

</comment><comment><USER10> Don't know about this one... Most tech startups were started out by techies.

I'd say it could work for non-technical startups.

</comment><comment><USER11> Excellent! Creativity doen't stop at the techies's desks.</comment>


Google's Project Zero researcher discovers "major" security issue in LastPass - <AUTHOR> http://www.independent.co.uk/life-style/gadgets-and-tech/news/lastpass-hack-security-problem-password-manager-a7658806.html <comment><USER2> Previous discussion [https://news.ycombinator.com/item?id=13960097](https://news.ycombinator.com/item?id=13960097)

</comment><comment><USER3> If you use online password management for anything security-critical then you're a fool. It pains me to see Lastpass so readily trusted even by the HN community.

<comment><USER4> Agreed. A combination of cryptsetup luksOpen foobar && mount foobar && vim foobar/passwords.txt has always worked fine.

I suspect it has to do with the modernist fetish of convenience. Not having all your data synchronized to all devices at all times is apparently a fate worse than death.

<comment><USER5> I started using LastPass because I found that for all but my bank, Google, and Amazon passwords, I was using the same password on every other page. I've found that it's really great to just let LastPass pick a lengthy password for every new site I join and know that I'll still be able to log into it later from my phone or my laptop or my desktop without problem.

I get that it's got some serious security holes, but it's better than not using it, because if I don't use it then I'm just gonna start repeating the same username and password across sites again.

The enemy I'm fighting is my own laziness. I'm not choosing between "use LastPass" and "lock my passwords in an encrypted fileystem." I'm choosing between "use LastPass" and "use the same password everywhere," and LastPass is better than that.

</comment></comment></comment><comment><USER6> So if they want 2fa enabled and users to avoid browser plugins it inadvertantly suggests a vector to start looking at.

At a guess, an API vuln that issues a token of some description?

</comment><comment><USER7> LastPass tire fire continues to spread.</comment>


Sarah Palin tells court that email hacker disrupted campaign and personal life - <AUTHOR> http://www.guardian.co.uk/world/2010/apr/23/sarah-palin-email-hacker-campaign

<comment><USER2> So the maximum is "50 years in jail". The man is now 22 years old (younger at the time of the "crime"). The duration of Palin's campaign was no more than a few weeks. Even the duration of the resulting presidency would be no more than 8 years. Imagine putting a man in jail for more than twice his lifespan, over what is essentially a stupid prank. Just what "damage" could have been done to society, to justify the same maximum prison time as a serial killer?

<comment><USER3> 50 years is just the total of statutory maxima for the 4 different crimes he is charged with. News and the prosecution will usually go for the biggest mumber in sight, for shock value. I doubt he will get more than a few years at most, partly because he did not attempt to enrich himself.</comment></comment>


The Impact of the 'Open' Workspace on Human Collaboration - <AUTHOR> https://www.hbs.edu/faculty/Pages/item.aspx?num=54899 <comment><AUTHOR> This is the most relevant part:

> Open architecture appeared to trigger a natural human response to socially > withdraw from officemates and interact instead over email and IM.

As we all knew, all along.</comment>


Marketing with Asana: The Tools We Use - <AUTHOR> http://blog.asana.com/2013/07/marketing-with-asana/

<comment><USER2> (I work for Swiftype.)

Search analytics are a really powerful way to find out what your customers want.

Another one of our customers, SupportBee, wrote a guest post for our blog about how they use Swiftype's analytics to influence their product development: [https://swiftype.com/blog/swiftype-search-analytics-for- supp...](https://swiftype.com/blog/swiftype-search-analytics-for- supportbee.html)

Our customers love the reports we provide - but we are just getting started.</comment>


Show HN: A new way to diff websites - <AUTHOR> http://www.diffd.com/?ref=hackernews <comment><USER2> Looks cool. I wanted to build a service that would monitor an external web page for me and notify me about content changes. The problem was that it was constantly detecting changes with ads or menus or stuff not related to the content itself (which is what I was interested in). Didn't have time to get it going any further. One use case for me was to monitor the Washington State traffic laws (or all RCWs if needed) and get notified when changes were published (with a diff that I could actually read, as opposed to just the URL with the change on it). I've found over the years that the laws were being changed in very small ways (like a 1 character diff, like changing "2" days to "5" days, giving LEOs more time to file traffic tickets with the court, or just a few letters, like changing "may" to "must" (requiring judges to dismiss tickets not filed in a timely manner as opposed to giving them the option).

<comment><USER3> With CasperJS and a CSS selector it would be easy to do, a few lines actually: [https://gist.github.com/Ivanca/aef2e58dbbf9eb3e1bd4](https://gist.github.com/Ivanca/aef2e58dbbf9eb3e1bd4)

</comment></comment><comment><USER4> If you're not interested in this because it's SaaS, check out BBC's Wraith instead:

[http://responsivenews.co.uk/post/56884056177/wraith](http://responsivenews.co.uk/post/56884056177/wraith)

<comment><USER5> Wraith is a cool open source project that we played around with a bunch before starting Diffd. This is roughly what the same diff of Stripe as shown in our demo will look like with Wraith: [http://www.diffd.com/imagemagick_compare.png](http://www.diffd.com/imagemagick_compare.png)

</comment></comment><comment><USER6> Nice, I've written a few scripts to do things like this such as creating heat maps of areas of change between two sites and giving you the percentage of difference for CI builds etc...

Do you have any plans to open source this? That's the big factor for a lot of organisations decisions as to whether they'll make use of software or not. For example we have a policy that we won't rely on any proprietary software within our build / test pipeline.

<comment><USER5> We have been considering going down that route and releasing Diffd as an open source project while offering a hosted version that is the easiest way to do things in parallel and across multiple browsers and operating systems.

However, we expect that initially most will add Diffd as a manual step at the end of their pipeline. So, it's not like their ability to push code is completely dependent on a third party, it's just another check.

Taking a dependence on Diffd is going to be less risky than depending on a third party for your CI server like Travis CI, CircleCI or CodeShip.

<comment><USER6> We don't depend on a third party for our CI though? We use Gitlab-CI to build all our code / Docker images and run our tests which not only works very well but is also extremely fast.

</comment></comment></comment><comment><USER7> Congrats on releasing, I was doing a similar thing, with a 'backup' twist but never got to do the MVP [http://clicktwin.com/](http://clicktwin.com/)

<comment><USER5> Thanks lazyant. Clicktwin looks cool, I think people might find the instant deployment to s3 useful if they had a dynamic website that was falling over from high load.

You might also be able to achieve a similar sort of backup with Cloudflare.</comment></comment>


Meltdown Initial Performance Regressions - <AUTHOR> http://www.brendangregg.com/blog/2018-02-09/kpti-kaiser-meltdown-performance.html <comment><USER2> > Applications that have high syscall rates include proxies, databases, and > others that do lots of tiny I/O. Also microbenchmarks, which often stress- > test the system, will suffer the largest losses.

My team's RDS instances got hit hard with a 40% increase in CPU usage: [https://imgur.com/a/khGxU](https://imgur.com/a/khGxU)

<comment><USER3> > _Also microbenchmarks, which often stress-test the system, will suffer the > largest losses._

Tangent: this only seems like more proof that we should not rely on them as developers, no?

<comment><USER4> It's proof they serve a specific purpose. If the micro benchmark is especially affected by the effect you're trying to measure, it provides an upper bound of sorts for the effect's magnitude. This informs the rest of the analysis.

</comment></comment></comment><comment><USER5> The syscall in his benchmark made me laugh ([https://github.com/brendangregg/Misc/blob/master/s1bench/s1b...](https://github.com/brendangregg/Misc/blob/master/s1bench/s1bench.c#L124)):

      close(999);	// the syscall (it errors, but so what)

<comment><USER6> It's surprisingly difficult to come up with a syscall that is guaranteed to enter the kernel and come back out on a fast path. Old choices for this were things like getpid() or gettimeofday() which are now handled in the VDSO. This seems fairly clever, honestly.

<comment><USER7> getpid is only fast on ia64 according to vdso(7).

<comment><USER8> glibc caches getpid, so you'll only see the syscall happen after the cache gets invalidated by a call to fork, clone, etc.

</comment></comment></comment></comment><comment><USER9> An open question is still who should enable the mitigation. The risk-cost doesn't seem to fit too many scenarios.

Meltdown requires running native, intrusted code, and that doesn't apply to too many servers. While it may be possible to chain this onto another exploit, once an attacker has gained remote code execution, you have much bigger problems.

While meltdown is interesting, i wouldn't enable kpti on my database servers buried behind other network infrastructure.

<comment><USER10> > Meltdown requires running native, intrusted code, and that doesn't apply to > too many servers. While it may be possible to chain this onto another > exploit, once an attacker has gained remote code execution, you have much > bigger problems.

[https://react-etc.net/entry/exploiting-speculative- execution...](https://react-etc.net/entry/exploiting-speculative-execution- meltdown-spectre-via-javascript)

<comment><USER11> You realize he's talking about his servers, and the article is talking about Javascript running in the browser, right? The only things talking to his database server is his own server code via SQL queries; if there's any browsers and third-party Javascript running on that database server he's had a real security meltdown, of the kind that makes Meltdown irrelevant.

<comment><USER10> Yes? I'm just addressing his assertion that _" Meltdown requires running native, intrusted code"_.

<comment><USER12> He's not wrong, it's just that your browser does that automatically every time it loads a web page. That's an insecure-by-default security model and it's why people run JavaScript blockers like NoScript.

A server, however, is unlikely to do this, so the risk profile of something like Meltdown is much lower. It's not nil because if someone _does_ get on your box, whether they're supposed to be there or not, Meltdown can be leveraged to read the memory anywhere on the system, effectively neutering the protections of a multi-user operating system. This is a major information risk by itself since it could leak important things like encryption keys or user login info, but it could also be used to make local privilege escalation exploits simpler. (That said, if a remote user is able to get a shell on your box and submit code to the CPU, it's probably not likely that you won't already have a latent local escalation vulnerability that can be chained to get root.)

So he's right that people who don't execute arbitrary code on their CPU are much less vulnerable, but they're not totally invulnerable because you can't really guarantee that no one will be able to submit instructions to your CPU due to RCEs and the like. Also, you essentially need to trust everyone who has shell access to your server with anything that may be held in the server's memory, including keys, passwords, etc.

People who are considering disabling KPTI will have to decide whether they want to make the extra local attack surface available in exchange for the performance gain under normal operation.

</comment></comment></comment></comment></comment><comment><USER13> What syscall rates do different databases sustain at maximum load? Transparent huge pages negating most of the overhead is very good news-- but probably helps less with mmap'd IO which so many databases use.

<comment><USER14> Mmap'd IO is still a shit show because of clearing the CR3 register on page faults.

<comment><USER15> What precisely do you mean by "clearing the CR3 register?"

<comment><USER16> Although slightly technically inaccurate, he clearly means a full TLB flush.

Needs to be done on Intel CPUs older than Haswell, on those CPUs without INVPCID support.

With INVPCID you can partially invalidate TLB.

<comment><USER15> Having recently rewritten Linux's TLB code, this is quite wrong. For an ordinary page fault, there's no flush at all -- changing a page from not present to present doesn't require a flush on x86. _Removing_ a page from page cache can be done with INVLPG, which had been around for a long, long time.

From 4.14 on, Linux has used PCID to improve context switches, independently of PTI. While writing that code, I did a bunch of benchmarking. INVPCID is not terribly useful, even with PCID. In fact, Linux only uses INVPCID on user pages to assist with a PTI corner case. It's not entirely clear to me what Intel had in mind when INVPCID was added

</comment></comment></comment></comment></comment><comment><USER17> It would be interesting to know the interaction between the patched host and the patched guest. As a simple example, if the host aggressively flushes the TLB, the performance impact on the guest of doing the same could be lower. On the other hand, depending on how the host was patched, the loss of performance of the guest could be different when using some features.

</comment><comment><USER18> Out of curiosity, why does the syscall rate scale descend from left to right?

<comment><USER19> He chose to graph from high syscall rate to low. I was initially confused, as most people would have shown the ramp up, left to right. Doesn't matter much though, once you get it.

</comment></comment><comment><USER20> It would be great to see performance deltas for AMD CPUs too, especially since Meltdown only effects Intel and AMD patches for Spectre Variant 2 are considered optional. I would also be nice to see a discussion of AMD's ASID and any differences it has with Intel's PCID when when PCID is addressed.

<comment><USER21> What's being tested here has nothing to do with Spectre. Only KPTI.

<comment><USER22> Maybe a better formulated question would be - do the Meltdown changes to kernel impose performance penalty on AMD processors as well (regardless of exploitability)?

<comment><USER23> They are not enabled for AMD by default. If you force them to be enabled, they impact performance (of course).

<comment><USER9> Even on Intel there should be an evaluation of performance cost vs risk. If you are not running unknown code, you may not want to enable kpti either.

<comment><USER23> Extremely correct. Database and frontend servers are hit pretty hard, nothing in the middle is hit. But neither of those ends is actually running untrusted code, for the most part.

I predict a decline of the "hyperconvergence" server and a return of the usual "database server" \+ "app server" \+ "frontend server" combo.

</comment></comment></comment></comment></comment></comment><comment><USER24> I'm kinda surprised Netflix weren't already using THP.

<comment><USER25> Not a Netflix employee, but at my last role we disabled THP on every instance we had. We had issues with databases (MySQL/Cassandra/HBase), Hadoop, and Java applications.

I hear it's gotten a lot better since then, and the compactor doesn't freeze stuff like it used to.

</comment></comment><comment><USER26> Do you have any idea how much it will regress your I/O workload on OCA?

<comment><USER27> The OCA team were working on it, but I don't have a number. My guess is small, since it uses sendfile and then the packets are all done kernel to kernel. So the syscall rate should be relatively low.</comment></comment>


Estimating the Effects of File-Sharing on Movie Box-Office - <AUTHOR> https://editorialexpress.com/cgi-bin/conference/download.cgi?db_name=NASM2017&paper_id=216 <comment><USER2> I wonder what effect it has on other forms of entertainment?

Suppose I'm looking for some entertainment this evening, and imagine piracy is not available, and imagine that there is a current movie in theaters that I'd like to see.

Let's say that tickets plus snacks would cost $20. If I decide that's more than I'd like to spend on an evening's entertainment, I'm probably not going to just sit at home doing nothing.

I might instead decide to watch a movie on pay-per-view for $5. There are plenty of movies that I did not deem theater worthy but that I would like to see, and would prefer not to wait until they show up on some channel that is included in my cable package.

Or maybe I'll decide the reading is the way to go for my entertainment this evening, and swing by Barnes & Noble or by Amazon's Kindle store and pick up a book and start it. Or maybe I'll pick up a magazine or two.

Or maybe I'll take a longer term view and decide to buy a game for $50. Yes, that's more than the $20 I don't want to spend on a movie, but the game will provide entertainment for several evenings.

Or maybe I'll go to a club to see a concert.

Now let's change things so piracy is available, and I decide to pirate the current movie. Since I would not have gone to the theater anyway, my choosing piracy does not effect current box office. But it does affect my spending on PPV or books or games or concerts.

I think to properly study the effects of piracy you may have to consider all forms of entertainment together as a single category.

<comment><USER3> Without money and piracy I may commit a crime to obtain money. I may get caught and have a crime record. This might force me into a life of crime.

To property study the effects you need to measure everything

</comment></comment><comment><USER4> People have a largely fixed entertainment budget. While it can flex some, it won't generally double for the majority of Americans, at least.

The number one impact on box office sales is a shitty experience. I would pay consistently for an Alamo Drafthouse type experience. I quit considering movie theaters, because I just don't feel good about any of it.

Very large numbers of my peers have done the same. Wait for the movie, buy a Disc, or download, make a great dinner, and it's a no contest choice. Costs about the same too.

As for piracy, funny how that works. I'm told it's costing billions, when I know better. I'm told it's theft when I know better too.

Buying media via download costs me roughly what physical media does, and I can't trade and share with friends. Not a huge bother, but yet another negative we don't need.

Honestly, I just found other experiences. It's hard to get me to even watch a movie today, and forget TV.

I just don't need it, and rarely want it. As a group activity, a movie still makes sense. I'll do that, but not in a poor environment, and I'm in no hurry.

</comment><comment><USER5> Noticed besides US, Australia and Sweden and Norway have high rate of file sharing? Wonder why is that? Taxes / tariffs on media?

Conclusions are interesting as well:

"File-sharing reduces total revenue of the motion picture industry from box- office by $ 90 million in total, 1.4% of the current box-office", and that's a lower estimate from what the industry claims.

<comment><USER6> Speaking as an Australian, the anecdotal effects I see as contributing to high piracy rates are:

* Online content is more expensive in Australia. This isn't due to taxes or anything, Australians just get charged more for the same products.[1]

* Content is very often not available legally in Australia by any means, or is delayed for months or years from the US release.

* (this one may be old, and is certainly actively changing) Poor internet speeds and data caps make streaming services not a viable option, whereas a download can be left running overnight so the final result can be stored and watched many times.

[1] [http://www.news.com.au/technology/australian-government- rele...](http://www.news.com.au/technology/australian-government-releases-its- report-into-it-pricing/news-story/d94bc5968b2dd690b2070cac903c3e58)

<comment><USER7> I think you have some causes and effect swapped. I suspect people pirate because some things aren't available in your area not the other way around.

There are tons of cases where video game companies do things like release a Russian translation on day 1 and for every other game from that studio it came two weeks later, then for that one title Russian piracy is almost non- existent.

<comment><USER6> > I suspect people pirate because some things aren't available in your area > not the other way around.

Yes, that's what I'm saying. Listed above are what I believe might be causes of high piracy rates, not effects.

<comment><USER7> Sorry, I misread you.</comment></comment></comment></comment></comment>


Ask HN: Urbit Sale – will you be buying? - <AUTHOR> Urbit ( https://urbit.org/ ), a peer to peer network of personal servers built on a completely new stack. The first sale of Urbit `Stars` (1/64th of the network) has just started, priced at 512 USD.<p>Have you been following Urbit? What are you thoughts on the sale, and will you be buying in or not? <comment><USER2> Background: I'm a decentralization researcher, I quit my job at Twitch to report on decentralized web initiatives full-time, and I'm tracking 200+ projects and protocols like this attempting to remake the web. Urbit is near the top of the list of ones I think will work.

Urbit has the most coherent vision for why decentralized computing is necessary across all other initiatives in this space. They understand the history behind topics of trust, identity, and governance in building new software platforms, in ways that other initiatives (especially blockchain- based communities) sorely lack, or are unable to communicate effectively.

Compare to Ethereum, which has a similar scale of ambition, but with a far less trustworthy and transparent leadership team, who has (despite these problems) still managed to raise huge interest in a currency with a several- billion-dollar market cap. If you just spend time reading Urbit docs vs Ethereum docs, I believe the difference in clarity of vision will become apparent to you.

I bought two stars and am considering buying a third.

<comment><USER3> I'm curious, what are the others at the top of your list? IPFS?

<comment><USER2> IPFS is certainly one of the top initiatives, and Juan Benet in particular will be involved in whatever future internet architcture takes shape. I need to investigate them more myself before I endorse the full vision, though.

</comment></comment></comment><comment><USER4> Unlike e.g. Ethereum back then, this still very much looks like an art project motivated by ideological/political reasons, so 95% chance to fail IMO. That being said the vision is very big. I still bought a star mostly to track it and a bit of FOMO.

<comment><USER5> Urbit is the most brazenly audacious computer science project on Earth, it's entirely open source, and yet, every time it gets mentioned on the world's most popular forum for programmers, virtually all of the discussion is non- technical.

Who cares about the ideology behind it? Does anyone pick their linux distro based on the maintainer's political blog? "Oh, one of the devs said something in 2007 about immigrants I didn't like, and..." uh huh. What about the insanely ambitious attempt to replace unix, any comments on that?

<comment><USER6> First, the person above you didn't say anything about caring about the underlying ideology; they said that they cared that the project seemed to them to be an art project designed for ideological purposes and unlikely to succeed as a result.

Second, people definitely do choose their linux distributions based on ideology. The Debian project, in particular, has a very well established ideology. They also choose which distributions they spend money on or donate money to (the relevant comparison to purchasing stars as discussed here) based on ideology.

Third, Urbit is a different category of thing from a Linux distribution. Urbit is a network. A network designed and organized based on a particular principle can impose that principle on the way that people on the network act, interact, and relate on the network in ways that a Linux distribution cannot. I am not certain whether this accurately describes Urbit. Statements from the project and its developers have suggested to me that this has been a goal, but they may have retreated from that since.

<comment><USER7> >A network designed and organized based on a particular principle can impose that principle on the way that people on the network act, interact, and relate on the network in ways that a Linux distribution cannot. I am not certain whether this accurately describes Urbit. Statements from the project and its developers have suggested to me that this has been a goal, but they may have retreated from that since.

this is certainly the case -- the scarcity of resources (about 4b cryptographic identities called 'planets', along with an associated reputation system) is meant to introduce something like proof-of-stake, where incentives are created that make it more expensive to act in bad faith (trolling[1], spamming) than the bad faith behavior is worth.

this is an interesting read the head developer wrote a few years ago about these subjects: [https://unqualified- reservations.blogspot.com/2010/03/future...](https://unqualified- reservations.blogspot.com/2010/03/future-of-search.html)

[1] 'trolling' in this case would be determined by local 'communities' on the network, and not a central authority a la the current internet (fb, twitter, etc)

</comment></comment></comment></comment><comment><USER3> I bought one star. I've been following the project for a few years and still have mixed feelings about it. The idea is awesome and the tech seems really interesting, but I always hit a roadblock with the intentionally opaque naming of things.

</comment><comment><USER8> A note for the future, if whoever posts this could please link the actual sales URL, it would help highly confused people. Like me.

There was no sale link anywhere on the site I could find (and I was there in the throes of the sale).

There was no mention of a need for an invite code to purchase a star. Or how to get one.

Even reading the documentation I'm still very hazy how a star is used other than real estate. There's no documentation about how a star code (assuming its a thing) is redeemed or associated or used.

The price points seem reasonable for the scale of what they're building. I hope they put some of the profit into better marketing and explanation coverage.

<comment><USER5> It was sale.urbit.org but you needed a ticket. Submit your email address to get on the list for future notifications, including tickets to future sales, although they say the next one will be a public auction. What you get when you buy is a number which can be submitted to urbit in exchange for the key that proves ownership of a certain star, meaning you would be able to sign messages that other planets/stars would trust as having come from the owner of that star. Stars are identical to planets except that they are also able to issue tickets for planets. So the point of owning a star is that you can give away or (in some future scenario where anyone wants to buy them) sell planets. For now, you can get a planet just by emailing the devs and asking for one.

People have complained about the documentation forever, but I think it's just hard to explain what it is, because it is pretty novel. If you chew through some of the available stuff and have further questions, reply to me, I'm as likely to give a useful answer as anyone.

</comment></comment><comment><USER9> Correct me if I'm wrong but I believe a star is 1/65536th of the network, not 1/64th. Perhaps you meant to say 1/64k?

<comment><USER7> well, they sold 1024 stars -- 1/64th of 65536

</comment></comment><comment><USER10> I was interested in bitcoin before it got really big but I was convinced by others that it was going to just fizzle out. Now FOMO is driving me to want to buy a star but it's much I'm less confident in this then I was of bitcoin.

<comment><USER11> I'm in the same boat. I heard about Bitcoin in the early days too, and didn't get involved because I was convinced it economically couldn't work. Did you buy a star? I missed the sale by a few hours, thanks to my hesitation.

</comment></comment><comment><USER12> I bought 2 stars, mostly because of my Fear Of Missing Out. My understanding of urbit is very superficial.</comment>


The Invisible Design Behind the Apple Watch's Many Faces - <AUTHOR> http://www.wired.com/2015/04/apple-watch-design

<comment><USER2> I can't understand how could the iwatch work when it's so fundamentaly at odds with the core philosophy of the object.

A watch is a symbol of keeping time, and as such is deeply linked to the notion of eternity. You may want to offer a beautiful watch to someone as a gift because it will last for his entire life. How many times in movies have we seen the dying father giving his watch to his kid.

That's a reason why swiss watch makers are able to repair your 50 yo watch, should it break.

Now, even with the best effort, we _know_ that the iwatch battery will wear off in two years, that the watch os won't be upgraded anymore in 3 or 4 years, and that the watch itself can't work if not linked to a device that also has a very limited lifespan.

Apple should have made the inside of the watch upgradable and advertize this fact. They change you the chip and the battery, but you keep the outside untouched. This way, you won't have to throw your _very personal_ device to the trash every 3 years.

<comment><AUTHOR> You have to understand that this device is not a watch, in the traditional sense its simply an iPhone Accessory which is named the "Watch because its worn on the wrist, same with all other smart watches none of them are watches they're jut phone accessories So to expect the kind of traditional sentiment that you get from a really beautiful well made swiss watch that lasts for generations is futile.

<comment><USER2> And yet, everything i read about this product including this article, emphasis finition and crafting quality, as well as the numerous watch face, and not new revolutionnary functions.

Apple clearly wants to make you believe it is a watch, and that's why the called it watch instead of Band, or any other neutral names.

<comment><USER3> I think many companies call it a watch is because, at the moment, the only things that do not look weird on anybody's wrist in daily life (1) are watches and bracelets, and a watch is the only one of these with a display.

(1) heart rate monitors don't look weird during exercise, but aren't worn all day long by any 'normal' people.

</comment></comment></comment></comment><comment><USER4> Apple seems to trying much harder to market the Apple Watch. I've never seen Apple being so open and giving so many journalists inside access.

</comment><comment><USER5> Thank you, I've been looking for another excuse to wholly submit to a marketing campaign.

<comment><USER6> Thank you for insightful comment

</comment></comment><comment><USER7> For the watch to work properly it needs to be paired with an iPhone, right?

When are our future wrist-communicators coming?

<comment><USER8> Apple Watch v2 with an inbuilt phone and SiriNext peronal agent will probably be a great device that many will buy, similar to iPad 2 that was superb compared to iPad 1.

</comment></comment><comment><USER9> Perhaps it is because Apple are such perfectionists, that when they miss some tiny detail it's actually quite glaring, because you're not used to them missing that kind of thing.

For example, the Finder's tags configuration dialogue is a masterpiece of horrendous UI design. It's the kind of thing you'd never expect Apple to get wrong, given their usual attention to detail: [http://arstechnica.com/apple/2013/10/os-x-10-9/8/#tags](http://arstechnica.com/apple/2013/10/os-x-10-9/8/#tags)

<comment><USER10> Apple are portrayed as almost infallible when it comes to design. They obviously put a lot of effort into getting things right but I've encountered many, many bad design choices in OS X.</comment></comment>


How to Tell Google News to Split Technology into : Tech, Autos, Games? - <AUTHOR> <comment><AUTHOR> Reading Google News, Technology is nearly useless these days. Games and Autos (while legitimate topics for sections of their own) so dilute "pure" technology topics that important items are not included.</comment>


The New Coronavirus Growth Debate - <AUTHOR> https://www.bloomberg.com/opinion/articles/2020-03-03/how-fast-will-the-new-coronavirus-spread-two-sides-of-the-debate <comment><USER2> Here's the real source of the debate:

"I look out the window and nobody has the virus. Nobody I know has the virus. I don't know anybody who is talking about anybody they know that has the virus. Virus? What virus? Oh, you mean the virus in China, the one sitting next to the their weird government and the air pollution. I guess that could make it harder for McDonalds to source their happy meal toys. Hey, did you hear about the latest thing that happened in the primaries?"

It's weird, and it may even be psychologically impossible, to look out the window at a sunny day and imagine that there's an explosion happening. You can look at the numbers, but even then, can you really believe it on a deep emotional level?

<comment><USER3> The problem is that statistically the confirmed number of cases is only 1 in 2 million in the US at the moment. Your chance of getting hit by lightning in any one year is higher at 1 in 700k. So at this stage it is very hard to comprehend.

It's when you consider the prediction that 40-70% of the adult population are going to be infected if it's not halted that you have to take it seriously. Noah Feldman while interviewing Harvard epidemiology professor Marc Lipsitch who gave that figure admitted that it just didn't seem real that we are looking at 1 million+ deaths in the US if that level of infection occurs. Very sobering thought.

<comment><USER2> As far as policymakers are concerned, it's the 40-70% that needs to be focused on as preparations are made. We're talking a few, maybe three months before this thing has blown up from an insignificant 1 in 2M and impacted most people's lives indirectly. What people have a hard time imagining is that something can go from insignificant to significant so quickly.

</comment></comment></comment><comment><USER2> > _But many medical professionals think in terms of what are called "normal" > statistical distributions. If someone visits your office with what appears > to be a typical flu case, it is usually exactly that._

These two sentences are completely, absolutely unrelated.

<comment><USER4> Obviously, it's a flawed argument, because the x-axis in this distribution would make no sense.

...but you clearly know what they are _trying_ to say: Among the population of people today with flu symptoms, the vast majority of them are influenza and not covid19.

I wonder if this is still true in places like Wuhan. I also wonder if we'll see a decrease in influenza next season due to the all the people adopting far better hygiene habits this year.

</comment></comment><comment> tuna-piano For the people who think it will grow exponentially (seemingly most epidemiologists), China's job in taking an uncontrolled outbreak and at least temporarily controlling it seems quite surprising. How was China able to control it, and why can't that happen in other countries? Their efforts are really quite extraordinary.

"I think people aren't paying close enough attention. The majority of the response in China, in 30 provinces, was about case finding, contact tracing, and suspension of public gatherings — all common measures used anywhere in the world to manage [the spread of] diseases."

(The above quote and both of these excellent articles are from the same primary source, Bruce Aylward from WHO):

[https://www.nytimes.com/2020/03/04/health/coronavirus- china-...](https://www.nytimes.com/2020/03/04/health/coronavirus-china- aylward.html)

[https://www.vox.com/2020/3/2/21161067/coronavirus- covid19-ch...](https://www.vox.com/2020/3/2/21161067/coronavirus- covid19-china)

Will this be enough to pause it for a long period of time? Or will it come roaring back once china lets their foot off the brake? Or will China extinguish the virus in their own country and have to block travel inbound to keep it that way? We shall see.

<comment><USER2> If you go around welding people in to their apartments, well, the virus will probably slow down.

<comment><USER3> Were this the zombie apocalypse, people would be signing up left and right. Corona virus carriers just don't have the same menacing appearance as zombies, so everyone's still blase like it's climate change (which has a greater potential to kill more people - just not right now).

</comment></comment></comment><comment><USER5> Rather, people should adopt a S-curve mindset, where the shape of the curve depends on the effectiveness of mitigation strategies.

</comment><comment><USER6> How about evolutionary growth model:

\- severe cases are quarantined

\- mild cases keep spreading

\- evolutionary pressure causes the virus to become less deadly

\- deadliness drops to flu levels at which point it stops being actively tracked

<comment><USER7> Alternatively it does not spread effectively enough (not enough reservoirs in summer) and dies out in human population.

Or widespread herd immunity reduces its severity, not mutation. (e.g. Polio) In the latter case, we would have some slight persistent number of cases.

</comment></comment><comment><USER8> What's the false positive rate of the test?

The UK has test 16,659 people and confirmed 85 of them, ~0.5% of those tested, as infected.

How many of them are false positives, if any?

<comment><USER9> (I don't know, and have been trying to find the answer, and have failed.)

Best I can do: Assuming that tests performed in different countries are comparable [1], your statistic would set an upper bound on the false positive rate, which could then be applied to other countries. The U.S. has performed 3.6k tests and has ~150 confirmed cases. If the false positive rate is the same, the vast majority of those cases have to be real.

[1] Big assumption.

<comment><USER8> The assumption may be wrong:

> The rollout of a CDC-designed test kit to state and local labs has become a > fiasco because it contained a faulty reagent.

[https://www.sciencemag.org/news/2020/02/united-states- badly-...](https://www.sciencemag.org/news/2020/02/united-states-badly- bungled-coronavirus-testing-things-may-soon-improve)

</comment></comment></comment><comment><USER10> They're both right. We are in a phase of exponential growth. If we don't check it it will absolutely overwhelm the medical system. But eventually we will have crossed the peak and reach a level more like a base rate. I don't get the "base raters" position here, it seems they're living two years in the future rather than concerning themselves about what happens on the way to that point.

<comment> tuna-piano If I understand "base raters" correctly, they are not saying anything in particular about this virus, just that in general due to all available factors (including human+government intervention), the world generally stays close to the same.

A month ago, if you asked a base rater and an exponentialist to make guesses on how many cases in China there would be today, I think the base rater would be much closer to actuality than the exponentialist (after all, their new daily case volume stopped growing almost a month ago!).

China was able to pause this outbreak, maybe they can hold it, maybe the rest of the world can too. I'm not holding my breath but it does seem like a possibility.

<comment><USER10> I see. That seems useless then isn't it?

I mean its fine if you're looking at the longer term picture. But its completely useless for informing how to handle an emerging situation. E.g. if China's decision making was informed by base rater thinking, which amounts here to "eventually things will return to normal" then they would not have taken the measures they took which are leading to that.

<comment> tuna-piano I'm just guessing too on what base rater is, but if I'm correct on what a base rater is, then the thinking has use for how normal citizens should prepare (maybe don't sell all your investments to buy rice and beans and N95 masks) rather than what those fighting the virus should do.

If my understanding is correct, the base raters don't say that nothing extreme ever happens, they only say that things usually turn out close to how they have been. I suspect that is true.

</comment></comment></comment></comment><comment><USER11> But is the number of cases really growing exponentially? The logarithm graphs over at [https://www.worldometers.info/coronavirus/](https://www.worldometers.info/coronavirus/) seem to indicate otherwise but maybe that's just China's successful containment efforts skewing the data?

<comment><USER2> According to the WHO, China is running a massively successful quarantine campaign at the expense of pausing their entire country. However, cases outside of China are still growing exponentially. Check out that plot - it's a straight line on a log scale.

<comment><USER12> > However, cases outside of China are still growing exponentially.

Arguably because we're still catching up with the initial wave of outbreaks that could develop unhindered. (as was the initial wave in China).

It would be very surprising, based on total population and density alone, if any other country sees more cases than China with maybe the exception of Iran which appears to be in disastrous shape.

</comment></comment></comment><comment><USER13> Getting data from [https://gisanddata.maps.arcgis.com/apps/opsdashboard/index.h...](https://gisanddata.maps.arcgis.com/apps/opsdashboard/index.html#/bda7594740fd40299423467b48e9ecf6)

My gut feeling is that far more people have this virus than the Johns Hopkins app says.

The biggest outbreak outside of China seems to be Iran, with 2,922 confirmed cases, 92 deaths and 552 recovered. This does sound bad since that is technically a 3% death rate, but my gut feeling is far more Iranians actually have it, but are not checking themselves into hospital because they don't want to be put in quarantine and would rather just stay home.

I'm not trying to downplay the seriousness, Corona virus still is something you don't want to get, but it probably won't kill you if you're a reasonably healthy person.

Unfortunately it will probably sweep the world before a cure is found, so we'll all be exposed to it eventually.

<comment><USER14> Except for some of us who come from the time you had to get permission to get an ID on the Arpanet. In that case, the WHO is not saying mathematically nice things about us. :-(

</comment></comment><comment><USER7> Why would anyone even remotely skilled in math consider a viral spread exponential rather than logistic or Weibull?

The former stems from communication theory and the observation that few people can be sick forever or more than once. There are also constant factor communication "costs" to the virus expressed as likelihoods of infection due to minimum effective viral load, countermeasures, severity and immunity.

Latter if fomites are a major factor in final spread. (overlapping point scatter gets you Raleigh - you have to be unlucky to be near the source)

Potentially a combination of both. They all look sigmoid.

The stronger the countermeasures, the lower the total maximum.

</comment><comment><USER15> Non-article. Tl; dr: Some people think corona is going to spread a lot, others don't.

</comment><comment><USER16> I get the feeling that its a double edged sword. Do you close schools(and more) to prevent the disease from spreading, or just overload hospitals with more patients? Either way, its going to hurt the economy. Which is better choice - is still being decided.

The problem is quarantine is not enough. We cant test fast enough - or at all. So is the only choice to keep people from gathering?

<comment><USER3> Quarantining the most at risk and those who look after them could be the best solution as far as reducing deaths.

They keep saying don't wear facemasks but if you are a asymptomatic carrier a facemask will limit you spreading it to others. And if everyone wore a mask correctly you then have two barrier protection. The problem is we don't have enough masks, and would quickly run out of them for health workers dealing with confirmed cases.

<comment><USER7> The problem is that _asymptomatic_ carrier is everyone. Face masks are not free yet, nor comfortable, not everyone will want to wear them, much less correctly.

</comment></comment></comment><comment><USER17> This guys understands exponential growth! Outstanding...

</comment><comment><USER18> The closest historical match is probably the 1957-1958 Asian flu epidemic. Airborne spread. There were complications in 3% of cases with 0.3% mortality. 68,000 deaths in the US, 2 million worldwide. Stopped by a vaccine developed after about a year.[1]

Live graph and map of current epidemic: [2]

[1] [https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2714797/](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2714797/) [2] [https://hgis.uw.edu/virus/](https://hgis.uw.edu/virus/)

<comment><USER7> We do not know mortality of this coronavirus truly yet, much less number of cases. It seems to spread somewhat less easily than flu, similar to other coronaviruses already known - like about 15% cases of cold. Potentially spread also shows seasonal nature.

I'd like to see rates and complication rates for common cold caused by coronaviruses.

</comment></comment><comment><USER19> [https://archive.md/yd4me](https://archive.md/yd4me)

</comment><comment><USER20> I'd put myself in the growther camp. In comparing it to ebola and SARS I think it is very different. With ebola the rate of death was very high, and very fast. It couldn't spread quickly due to areas of infection dying out quickly, plus it was blood borne and the Congo is not hyper connected to the rest of the world. For SARS the world got lucky there, again it had a higher rate of death than COVID-19(10% mortality vs. ~3%) but China back then(2003) was a different beast than China 2020. Its a lot more tied in to the global economy and overall now has a higher standard of living ( more tourism, more business flights).

For COVID-19 which is extremely concerning, based on current medical knowledge is that people without symptoms can transmit the virus, and that the virus could take 14 days to present symptoms. Looking at exponential growth in cases at South Korea and Italy show how fast its spreading. Good news is governments are seeming to take it seriously and remdevisir and a few other drugs looks promising to fight it given a vaccine is a ways off.

</comment><comment><USER21> Pointless article. Posits people fall into two camps, the smart people who understand exponentials, and idiots who don't. Guess where the people in Washington, D.C. fall.

<comment><USER22> Not at all. If you know the context of superforcasters (See Good Judgement Open) they are the only ones who are consistently good at making forecasts. Even most experts fair no better than chance. And they rely on detailed assessments with a combination of different sources and lots of analytical work.

<comment><USER2> But what's the base rate here, the base rate of years that have pandemics, or the base rate of exp(t) functions that go up really fast?

</comment></comment></comment><comment><USER23> It's a good point, HN thinks after 60 years of sucking and being useless Augmented Reality (AR) is going to go exponential this year!

But then someone in the Valley did beat a hell of an old Taxi institution exponentially.

</comment><comment><USER24> Meanwhile the trump administration is hiding statistics and desperately trying to privatize testing as fast as physically possible so the test counts/result counts are not available as public analytics.

Surprised few have noticed this. It's a bit insidious because the cases are going to explode.

<comment><USER2> Forcing every test to go though the highly limited CDC was a terrible policy move and was definitely worse than letting the hospitals do it. Trump has some blame here because it should have been changed sooner, but letting hospitals run tests is likely to increase the number of known cases. The former (under Obama I think) FDA head was against the ban that Trump lifted.

</comment></comment><comment><USER25> Very accurate division between the two mindsets. Considering the overwhelming base-rate bias, the contrary growth bias is the one that is in need of a larger voice.

Also, there's some reasons to think this may be different than these other outbreaks. One reason is that plenty of smart people suspect this was an accidentally released bioweapon, given the ground zero was so close to China's only bioweapons facility.

<comment><USER26> 1) no they don't.

2) even if they do, that doesn't mean a single thing without any kind of supporting evidence.</comment></comment>


Cops Would Be Liable Arresting Citizens For Recording Under Approved Conn. Bill - <AUTHOR> http://www.pixiq.com/article/connecticut-senate-approves-bill

<comment><USER3> _Looney also noted that interfering with an officer is already a crime and would remain so._

The thing that always bothers me about the anti-photo/video side uses the bogus argument that people taking photos or videos will interfere with police business. If the photographer is being a material interference, they are committing a crime with or without their camera. Otherwise, just let them be.

<comment><USER4> > he anti-photo/video side uses the bogus argument that people taking photos > or videos will interfere with police business.

that is a common pattern in authoritarian regimes -- a slew of catch-all laws that can be applied selectively. there is "interfering with police business", "interfering with official business", and whole set of others that can always be used to at least get you to the police station.

</comment></comment><comment><USER5> Per the article, LEO has immunity:

"if they were acting within their scope of authority and the conduct was not willful, wanton, or reckless."

I think this is pretty much the way it works everywhere in North America - sworn officers are ( rightfully ) immune to prosecution while in discharge of their duties. _unless_ their conduct qualifies as malfeasance. Arguably, ( in my mind at least ) arresting people for simply recording the public conduct of an officer already falls into this territory.

<comment><USER6> The article does not say that they have immunity in that case. It says that can be repaid for any damages that they end up owing.

</comment></comment><comment><USER7> Your all a bunch of morons. Since when in a civilized society to you think it is okay to belittle a guy for doing his job? By your description, he was simply looking at you and did not say anything. You are both allowed to look at each other. Your interpretation of his look is just your opinion but it sounds to me that you were the confrontational one. In what world is it okay for you to challenge anyone about meer look and reply with obscenities "What the fuck are you thinking?" I would have to say that this security guards suspicion about you was dead on. Get a life please and do not waste anyone's time again with your stupidity. I'm sorry I took the time to read your post and am now dumber myself for it.

</comment><comment><USER8> Tangentially related personal anecdote:

This morning on my walk into work, I passed two private security guards standing in front of Bank of America on Market Street [1].

I noticed they were standing on little rubber mats to reduce the strain of standing there all day - as I looked at one, I glanced to the other to see if he also had a mat and see if it was the same size as the others'.

As I did so, the short asian guard said very intently at me, in a way he was trying to assert his authoritative role, and said gruffly "GOOD MORNING" and glared at me as I walked by - so I continued staring at him.

I said back to him "What? Do you think I am not allowed to look at you now? What the fuck are you thinking"

He glared at me with his hand on his gun.

I wanted to go back and take his picture, but it would certainly aggravate this already moronic figure with an authority complex.

[1]: [http://maps.google.com/maps?q=one+market+sf+ca&hl=en&...](http://maps.google.com/maps?q=one+market+sf+ca&hl=en&ll=37.793953,-122.395458&spn=0.001202,0.002642&sll=35.746512,-92.109375&sspn=129.750612,143.789063&hq=one+market+sf+ca&t=m&z=19&layer=c&cbll=37.793921,-122.395497&panoid=4o82tBocunJTbJLY9EANww&cbp=12,104.59,,2,5.06)

<comment><USER9> >As I did so, the short asian guard said very intently at me, in a way he was trying to assert his authoritative role, and said gruffly "GOOD MORNING" and glared at me as I walked by - so I continued staring at him.

If that were my job -to guard a bank or whatever, I too would eye everyone with suspicion, as my default behavior. If I were a greeter, on the other hand, I would trust everyone as my default behavior.

Maybe it's just me, but if I were getting paid to guard, I would at least try to do so. I don't mean I would be a dick to everyone, but I would certainly "watch" as many people as walked into my field of vision.

I'm not sure acting relaxed and friendly and being off-guard would serve me in the capacity of being a guard. (Altho' I'm open to being corrected, if this assumption is incorrect).

<comment><USER8> If being looked at by a well dressed business man sets off your alarms as though you believe you are being cased, to the point where you feel you need to have your weapon hand at the ready - then screw that.

Screw this guard and every single mentality that supports him and people like him.

I am not a fan of the police state - and the last person I have any respect for is an armed civilian who believes himself to be some arm of the law.

Simply put, he is a hired thug for the bank. I give him no greater mind, or respect because he is wearing military like clothing. Here I went back and took this picture of him:

<http://i.imgur.com/E0rjw.jpg>

 __edit: and to be clear, if I were casing him - he would never know it. If I had bad intentions for him - clearly I would have used a moment like this where he was texting on his phone and oblivious to those around him to spring Operation Currency Liberation.

<comment><USER9> >If being looked at by a well dressed business man sets off your alarms

If I were a guard, I would not make any assumptions about how people are dressed.

>to the point where you feel you need to have your weapon hand at the ready

I hear you. If I were in your shoes, I too would feel like there was an overreaction. I do some street photography on occasion, and I do get weird looks from guards, on occasion. When you do photography a while, you learn to brush it off. Also, If I were the guard, I would remind myself that an unprepared guard is a potentially neutralized (or dead) guard.

We don't live in Japan where you can pretty much assume, that with very rare exceptions, a firearm will not be produced by an assailant.

>and the last person I have any respect for is an armed civilian who believes himself to be some arm of the law

Ok, we agree there.

>Simply put, he is a hired thug for the bank.

That statement tells me a lot. It's a pretty antagonistic interpretation of his aggressive attitude. Maybe he is maybe he isn't. Do we know he has a criminal record of unprovoked violence toward innocent people?

> if I were casing him - he would never know it.

I'm pretty sure you would not let him on. Do we know others would not be more impulsive?

For all I know he is a dick and does explode violently at others unprovoked. Still, I didn't read any evidence to that effect.

<comment><USER8> > _It's a pretty antagonistic interpretation of his aggressive attitude_

I find it ironic that his overly aggressive attitude makes me antagonistic, but I will concede that i get more emotionally heated about such things because i have contempt for his delusion of authority or power.

> _I do some street photography on occasion_

A while back, while doing some street photography of a homeless woman feeding some pigeons, in an alley not far from where this guard was, a guard emerged from some building telling me I was not allowed to take pictures.

I challenged her saying 'why?' - she said it was a federal law. I asked her if she was a federal employee or guard. She reiterated it was 'federal law' - <USER2> said 'but the building behind you isnt a federal building.'

She tried to press her position. I simply said "I don't care if it is a federal law - I refuse to obey it, further, you're not a federal employee and this is not a federal building" I also said "If I am standing on public property, and I can see it, I am allowed to photograph it. Period."

She realized she wasn't going to get me to stop, and left.

<comment><USER9> In talking to other photographers what I've learned is that there are two different approaches to take upon confrontation: One, defend oneself about extant rights as if life depended on it. Two, take a more practical approach, explain oneself (even if this should not be necessary) and if their objections remains, back down calmly. Fight another day.

While one would prefer that people not try to exert power where none exists, life will proceed smoother, if approach two is taken (as it regard photography)

Approach one can lead to agressive behavior by [guard|police|public] and has resulted in detention/citation. Approach two can and has resulted in some understanding and acceptance.

Confrontation one: You have no right to inquire about what I'm doing while on a public street. You ignorant $%#^# I'm going to call my lawyer, I'm going to sue you for harassment, etc...

Confrontation two: Are you familiar with street photography? It has a long tradition. You might have seen some of the photography on gallery visit. Have you heard of Winogrand, Cartier-Bresson, Frank? Here, I have some examples --carry some print-outs or load some into your DSLR.

</comment></comment></comment></comment></comment></comment><comment><USER10> "...Under the bill, officers cannot be found liable if they reasonably believed that the interference was necessary to (1) lawfully enforce a criminal law or municipal ordinance; ..."

Is there anything that would prevent a police department from convincing their municipality to enact an ordinance stating that recording of police officers was illegal?

<comment><USER11> I didn't see a specific supremacy clause in the text of the bill, so probably that will happen, then the state will have to go back in a couple years and amend this to say that the state law overrides any municipal law.

</comment></comment><comment><USER12> Proud of my home state for once.

</comment><comment><USER13> Does this mean you no longer need a permit to film in Connecticut? Ought to bolster the movie industry there.

<comment><USER14> If you are making a documentary about the police doing their jobs, then I think you can go ahead. Not sure if you need permission from documentary subjects to show the film, but you can film it.

</comment></comment><comment><USER15> The more power you give someone, the more you have to keep an eye on them so they don't abuse that power.

So I hope this spreads to the other states. It should be a federal law. All citizens should have the right to record police officers in public. We pay for them. They're job is to protect us. They are given a lot of freedom and power. And if they step over a line they should be punished the same way we would if we step over a line. Besides, photographing or recording people in public is perfectly legal.

I always thought it was stupid how Britney Spears can be hounded day and night by paparazzi or the news can play your frantic 911 call, violating your privacy when you most expect and need it, yet in Illinois you record one officer while he's giving you a ticket and you get sent to jail with rapists, robbers, and crack heads.

</comment><comment><USER16> Does anyone have stats on how many people are arrested for recording police?

Why (and why) did police begin to think that they could arrest people for taking pictures anyway?

<comment><USER11> No idea on the stats, but there have been a few high profile incidents in the news in the last several years.

For awhile police officers have been attempting to apply a personal right to privacy to their on-duty activities.

<comment><USER16> Yes, those high profile cases always get my blood boiling - far more than the general public's blood, apparently.

My question was _supposed_ to be "When (and why) did police start arresting people for photographing them?" and while I don't have data to support this, I think it was a combination of a) Rodney King, and b) 9/11. The former incident demonstrated the power of personal video recording in creating problems for police. The latter incident gave police more confidence in an broader range of powers, including the tangentially related issue of enforcing ban on the photography of sensitive places (which is, of course, a rather stupid policy, but I digress).

So, greater awareness of risk plus enhanced power yielded the application of that enhanced power to reducing that risk. Could there be anything more reasonable, more human, and more evil?

</comment></comment></comment><comment> excuse-me I may be slightly confused here. If it is legal to do something - take a picture - then why do you need a specific bill to allow you to sue a police officer for arresting you for doing something that isn't a crime?

Doesn't this get a bit infinite-set? Do you need a specific bill to allow you to sue a police officer who arrests you for doing anything else that is allowed in the Constitution?

<comment><USER6> The difference is that here you are able to sue the police officer rather than the police department, as you usually would. The theory is that this will discourage officers from acting in a certain way, since they would be personally liable.

However, one could also argue that this is bad for victims, since the officer's pockets will almost always be shallower than the municipality's. However, if this law were to pass, a smart attorney would probably file a suit against both the municipality and the officer.

<comment><USER17> One could also argue that it's bad for victims of other crimes. If police are personally liable for everything, then I suspect the won't be particularly motivated to do _anything_.

These ideas tend to come with unintended consequences.

<comment><USER18> Oh please, the sky isn't falling. Rape is not going to go uninvestigated because a police officer can't tell someone to stop filming.

<comment><USER17> That's not what I mean. Boil this down to it's logical conclusion. If a cop arrests anyone for something that is deemed after the fact unjustifiable, they should similarly be held liable for damages. I mean, why stop at taking pictures? Is it not just as much an inconvenience to be arbitrarily arrested for burglary?

Throw in the culture of the police corps - and you could see a lot more "not my job" type behavior from the police. This sort of law could very well be interpreted as a slap by an ungrateful populace.

I'm just saying there is a balance here. Us vs. them isn't likely to result in improvement. This law seems very knee-jerky to me.

<comment><USER18> Don't change the subject, it's exactly what you mean. This law doesn't cover "anyone" for "something," it's covers photography, period.

<comment><USER17> You're missing the bigger point. Look at the message sent. It's bigger than some hippie taking pictures at an occupy berkeley event.

<comment><USER18> No, I'm not. I simply don't agree that there is a "bigger point" at all, nor that this is a slippery-slope that must be guarded.</comment></comment></comment></comment></comment></comment></comment></comment>


Code Climate Now Supports Pull Requests and Branch Comparisons - <AUTHOR> http://blog.codeclimate.com/blog/2013/11/06/code-climate-for-pull-requests/ <comment><USER2> I've been waiting for this for a long time. The implementation looks great. Thank you!</comment>


Analyzing networks of characters in 'Love Actually' - <AUTHOR> http://varianceexplained.org/r/love-actually-network/ <comment><USER2> As a side note, the blog poster, and the blog post he references, characterizing the movie as terrible annoys me and shows that they both absolutely missed the point of the movie.

The movie depicts all sorts of different types of "love". Lust is characterized by the man who cheats on his wife; Sex is characterized by the guy who goes to the US; Platonic Love is the love between the rock star and his manager; First Love is the story of the young boy; Unrequited Love or Unattainable Love is the man who is in love with his best friend's wife, and so forth.

It's not a terrible movie at all, it's freaking brilliant.

<comment><USER3> It's a pretty screenwriting 101 level of symbolism. Why does that make it brillant?

<comment><USER4> The poster successfully conveyed the insight that made it brilliant film for them and their enthusiasm makes me wish to see it again.

I don't know what practices are actually taught in 101, but the greatest artists won't be those that think themselves above them, but those who really appreciate them.

You wish to suggest you are superior but multiple stories to give different angles of a concept isn't symbolism, its a theme. The narrative structure is definitely unusual in the genre, frequently described as clever/innovative by reviewers and stands as a substantial achievement to have brought so many stories together successfully. None of that means you have to like it but the cheap middle-brow dismissal fools no-one.

</comment></comment></comment><comment><USER5> Seems like you could get new insights and ideas by getting lost in the graphs (if only I had better recollection of this film!) The interactive graph- timeline is a rather uncommon visualization technique, but I can see it being very useful. It only needs smooth graph transitions while using the timeline scrubber, and it could make a very useful tool with different applications.

And as films go, I think a similar analysis performed on Cloud Atlas would be extremely interesting.

</comment><comment><USER6> If you like that movie you must read through to the very last sentence of the post. It's a terrible pun? Joke? Reference? Whatever it is, it's funny and a great end to a nifty bit of analysis.

</comment><comment><USER7> A minor R tip: if you're trying to filter on substrings in dplyr, using grepl() is faster and does not require loading an extra package.

That said, this is a great, clear writeup, and I wasn't aware R had dendrogram capabilities _natively_. I'll have to spend more time looking into Igraph too.

<comment><USER8> I find that surprising. Do you have benchmarks?

<comment><USER7> Huh. Turns out I'm wrong. I ran benchmarking on a 1.8M row dataset of SF crime data w/ 100 replications.

Running grepl() via:

       benchmark(df %>% sample_frac() %>% filter(grepl("ARREST", Resolution)))

completed in 257.58 seconds.

Running str_detect() via:

       benchmark(df %>% sample_frac() %>% filter(str_detect(Resolution, "ARREST")))

completed in 248.917 seconds.

Overall, str_detect() is faster by 3%. The stringr packages has more surprises than I thought!

</comment></comment></comment><comment><USER9>
        For legal reasons I don't want to host the script file myself, but it's literally the first Google result for "Love Actually script." Just copy the .doc contents into a text file called love_actually.txt

Isn't this fair use?

<comment><USER7> The use of the script for statistical analysis is fair use; _distributing_ it isn't.

</comment></comment><comment><USER10> nice work</comment>


Ask HN: Can you really listen to tech podcasts while working? - <AUTHOR>

It just doesn't work out so I always ended up unsubscribing from most of the podcasts I once liked. What are the time slots you allocate to listen to podcasts? <comment><USER2> Not just tech podcasts, using my service podcastomatic.com (text-RSS-to- podcast) I convert all tech blogs into podcasts.

I listen on the commute and while coding. I assume no employer would object to their employees listening to tech podcasts as long as they are still doing their work.

<comment><USER3> That's so awesome. I had an idea a few years back that I called [in my head] "RSS Radio", and today I learned it exists. You're starting off my 2013 right. :)

</comment></comment><comment><USER4> yes. but it depends what kind of work, and what kind of stuff is being talked about on the podcast. I'll often pause the playback while I work out challenging parts of the project I'm on, or if I need to read something.</comment>


Can we save posts on Hacker news ... - <AUTHOR>

Really most of my tech news of the day comes through this site, and nothing would make me happier than to upvote a story and have it saved.  I know there is delicious and other stuff, but its just a pain to go to another service and save the link when 90% of the links come from this site. Just thought I should throw it out there and see what other people think. <comment><USER2> _nothing would make me happier than to upvote a story and have it saved_

Try clicking on your username in the upper left, and click on the number after "saved:"

<comment><USER3> Upper right.</comment></comment>


Guido van Rossum: Python 3000 and You - <AUTHOR> http://www.artima.com/weblogs/viewpost.jsp?thread=227041

<comment><USER2> The PDF of his PyCon Powerpoint presentation nicely summarized the switch to Py3k. I have been following it, but it was a nice reminder of just what is changing.</comment>


Uber and Lyft are being blamed for most of the traffic in San Francisco - <AUTHOR> https://www.kron4.com/news/bay-area/uber-and-lyft-are-being-blamed-for-most-of-the-traffic-in-san-francisco/1528303611 <comment><USER2> Lame article.

How have Lyft and Uber contributed to the drop in DUIs and drunk driving in general?

What about accidents? Insurance claims? Slower traffic was advertised as better for decades, for fuel safety etc.

What this shows to me is the city needs to figure out how to adapt it's relationship to cars in general.

I hate cars, the city should be thinking about how to eliminate the need for cars in it's limits as much as possible.</comment>