The Metric We Need to Manage Covid-19 - <AUTHOR> http://systrom.com/blog/the-metric-we-need-to-manage-covid-19/ <comment> dan-robertson If you look at test results you probably won't learn much. There will be far fewer tests than cases and most cases will never be tested. It seems more likely that you'll just be measuring some other effect of how testing is done, for example if the number of tests per day ramps up slowly (and non exponentially) then it seems likely that you will observe Rt to be decreasing (as the number of confirmed cases isn't shooting up), especially if people are mostly only tested when they turn up in hospital and are expected to be positive. I don't know how much the author's model accounts for that but I find the confidence intervals very unconvincing. I would expect them to be much larger.

This study tries to estimate Rt in several European countries from the death rates though there are issues with those statistics too: [https://imperialcollegelondon.github.io/covid19estimates/](https://imperialcollegelondon.github.io/covid19estimates/)

<comment><USER2> I saw an interesting alternate statistics where you measure the overall mortality rate and compare to previous years. EU only, but something like this might exist elsewhere as well:

[http://euromomo.eu/](http://euromomo.eu/)

The data lags behind a bit, but since it measures _total_ number of deaths, it doesn't matter if different countries measure coronoa-related deaths differently. Everyone measures actual deaths. The numbers are hard to fudge for politicians who wants to boost their results.

As a bonus, this also measures how your society is handling the pandemic overall. We know that people will die of the lockdowns as a second-order or third-order effect, and this way we can measure that as well.

<comment><USER3> How do you interpret a zero negative excess death rate? Sweden has over 1000 Covid attributed deaths, but almost no difference in weekly deaths compared to the same weeks in 2019. Death numbers are extremely noisy and especially in the flu season with the flu one year perhaps twice as deadly as the previous year.

Covid restrictions have also reduced (a _lot_ ) some other seasonal diseases like flu and noro/caliciviruses, so you get an effect in the other direction too. When those that die from Covid are in the highest age groups it's not unthinkable that a part of them would have been killed by the flu too, only the flu didn't come this year because of Covid. Perhaps that's why we have a lot Covid deaths but no excess deaths? Who knows. It's too noisy and has too many dependencies to be very useful I think.

<comment><USER4> Less people driving?

<comment><USER5> Also perhaps fewer elective medical operations, which result in fewer iatrogenic deaths (3rd leading cause of death, at least in the US).

<comment><USER3> This is a good candidate. Although some of this effect might be countered by people dying because they didn't get elective treatment.

Regardless: when you change pretty much every parameter of society (risk behavior, driving, crime, health care, ...) I don't understand how one can use the overall death statistics and try to attribute them to Covid.

<comment><USER4> I think the elective treatment theory is a better point than my driving guess. If you have a disease and your options are to get surgery or take medicine, you are probably being pushed towards non-surgical options, even if that would have overall worse results on average. This might ultimately raise the mortality rate, but would temporarily depress it.

Anecdotally, I definitely find myself trying to be safer right now. I have avoided ladder work on my house and been super careful when using a knife. I just do not want to go to a hospital right now.

I think looking at the overall death statistics are super interesting though. If traffic deaths fall, that is in some ways a byproduct of Covid. If more cancer patients die, that is also a byproduct. All of this goes into factoring how much damage and protection both the disease and the quarantine orders have caused.

</comment></comment></comment></comment></comment></comment></comment><comment><USER6> Jupyter notebook for this available here: [https://github.com/k-sys/covid-19/blob/master/Realtime%20R0....](https://github.com/k-sys/covid-19/blob/master/Realtime%20R0.ipynb)

</comment><comment><USER7> This site does this [https://epiforecasts.io/covid/posts/national/united- states/](https://epiforecasts.io/covid/posts/national/united-states/)

<comment><USER3> This site also uses "confirmed cases" instead of deaths. That would be useful many weeks ago, but now in most countries it's no longer a useful number. It should use deaths _and_ a country specific reporting factor for e.g. whether deaths include only hospital deaths or also other deaths, if only deaths with positive tests or includes assumed Covid deaths.

</comment></comment><comment><USER8> How can a confidence interval possibly be calculated without accounting for the number of tests—not just the number of positive tests—and the total population size?

What TFA shows us is the Rt _assuming that all cases are known_.

The model which knows nothing about testing rate, cannot possibly be used to predict R0 or Rt.

As a simple proof, consider what the model would output I f testing simply stopped. 0 out of 0 tests positive. After several days, the model outputs an Rt=0 with 100% confidence.

If you could account for testing rate, and then attempt to account for missed cases through proxies using hospitalization rates, ILI surveillance data, excess fatality, as well as incubation period, and an asymptomatic rate... maybe then you could draw error bars around an estimate of Rt that would pass the smell test?

If testing was run against 1,000 randomly selected people per geographical area each day, with a positivity rate reported over time, that would provide the perfect basis for making these calculations. The time series of test results that we actually have today require enormous amounts of unpacking before they can be used safely to extrapolate much of anything.

<comment><USER9> Rt is based on the rate of change, not the absolute value, right? So if testing coverage is constant in time, then doesn't it drop out?

Of course, it isn't constant with time, but it is also probably not changing that drastically over short time frames (like the seven days the author uses), so it shouldn't throw the results off too much.

<comment><USER8> Even if the number of tests stay the same, but just the testing criteria shift over time, it would invalidate the daily positive test count for use in determining Rt without somehow adjusting for the resulting bias.

</comment></comment></comment><comment><USER10> For lack of better charts. I have been calculating by hand the number of days since total cases(or deaths) were half as they are today. Example there are 586k cases today. How many days ago we had 290k ? Almost 10 days. So it is still doubling, but not as fast. A week ago it was doubling every 7 days.

<comment><USER11> I've been following the the death rate, because I think deaths are probably a more accurate statistic than cases - less likely that a covid19 death goes unreported.

This of course is a significantly lagging indicator, but it has been long enough that we can see social distancing is working.

<comment><USER12> > less likely that a covid19 death goes unreported

I don't think this is true at all. Many people die at home. Many die without having ever been officially tested. I've read that in some places the death toll is only counted from those who were tested and then died in a hospital.

<comment><USER13> It's definitely still "less likely" that a death goes unreported versus an infection without a death. Both numbers are shit, but deaths likely less so.

<comment><USER14> I think deaths might be unreliable too, because 1) we don't know the mortality rate, we're extrapolating it from bad data and then using an extrapolated number to calculate a trend and 2) mortality rate would change between different groups, e.g. sick and old people vs healthy people, depending how infections spread.

<comment><USER15> That's why we're talking about deaths, not mortality. Mortality requires you to know the number of people who are infected which as stated above can be inaccurate because many people are not getting tested. However, it's less likely that a person who dies of pneumonia in hospital won't be tested. Also, there is likely to be a smaller proportion of deaths happening outside of hospitals than there are infections so you should consider the number of deaths to be more accurate than the number of cases.

<comment><USER16> _> That's why we're talking about deaths, not mortality._

I believe you're confusing fatality and mortality. The correct version of that statement is: _" That's why we're talking about mortality, not fatality._

The case _fatality_ rate is the ratio of number of deaths to the number of people who are confirmed to have the disease (which depends a lot on standard of care, reporting, and bureaucratic integrity) whereas mortality rate is the rate of deaths in general.

</comment></comment></comment></comment></comment></comment></comment><comment><USER17> Without systematic random testing these numbers are describing testing activity, not effective reproductive rate.

Infection rate is not the same as number of cases.

</comment><comment><USER18> Again, much as these are pretty, without decent test data, they are a sideshow. The emphasis really should now be on proper sampling of a cross section of the population.

That should really be weekly testing of 1000 or more, over a range of backgrounds and wealth. Only then can we really see the impact.

Using mortality is only really useful when we know what the r0 is, or the mortality rate. We don't know either.

<comment><USER19> Looks like we're testing 10000 a day on weekdays, and over 5000 on weekends.

I find it difficult to believe with all that testing going on that no one is reserving 3% of the weekly tests to do proper estimation as you suggest.

[https://www.cdc.gov/coronavirus/2019-ncov/cases- updates/test...](https://www.cdc.gov/coronavirus/2019-ncov/cases- updates/testing-in-us.html)

<comment><USER20> Locally (upstate NY) per our pediatrician they're only testing people with symptoms _bad enough to put them at risk of hospitalization_. We're still very short.

<comment><USER19> Isn't that what you would expect any clinician to say, even if they were doing sampling? They're not going to use people asking for tests, they're going to select a group of people at random from the general population and invite them to be tested. Everyone else who has no symptoms they're going to politely say they're not testing people with no symptoms.

<comment><USER20> The point is that tests are still in extremely short supply, at least around here.

</comment></comment></comment></comment></comment><comment><USER21> Interesting article. In late March I built a simple SEIRD model for a few US states with python. It was very difficult to estimate R0. R0 is essentially Beta/Gamma, where Beta^-1 is is time between contacts and Gamma^-1 is the time until recovery. I think what should be added to this analysis is population density. Each state has a different rate that people come into contact with one another. It would make sense that R0 in NYC is much higher than in Iowa.

Also this website is crowdsourcing US COVID-19 mortality forecasts if anyone is into data modeling:

[https://www.unitarity.com/app/challenges/us-coronavirus- outb...](https://www.unitarity.com/app/challenges/us-coronavirus- outbreak/events/apr-20)

This website

</comment><comment><USER22> It's really too late to manage this. Testing hasn't been made available in time mostly due to total failure to plan and political decisions not to manage any of this.

All we can do now is turn up and down the level of isolation depending on how many ventilators and ICU beds are available. We will have to do that on a sort of feedback, trial and error basis because we lack capacity and what capacity we do have is being wasted.

The only question is whether anyone will be held accountable for this colossal mess and the preventable deaths its caused, or if we will just accept this happening ever 5 or so years...

<comment><USER23> Depends.

If active cases can be pushed down to a mange-able level (i.e. a level where contact tracing is a realistic option and outbreaks can be, by and large, contained) a return to relative normalcy could still be possible.

So that's another one, two months of tight lockdown to push down active cases (to, say, 500 in the NY area). Then it becomes feasible to throw massive amounts of people at those 500 cases and to do aggressive contact tracing and quarantining of all contacts.

Maybe just by people interviewing the infected, calling around, playing detective (a bit slow), maybe also with the help of contact tracing through mobile phones. Also, you obviously would need to be able to test for the virus (ideally) whenever someone shows the slightest hint of related symptoms (and since those are so unspecific and not exactly rare you would have to expand testing capability).

Some measures will obviously stay in place, but many of the most drastic and impactful ones could be reduced since there wouldn't really be uncontrollable community spread.

That, to me, sounds like a realistic plan (unlike a complete lockdown until then or trying to let the wave wash over us) for the time until we have a vaccine and until that vaccine is availible in sufficient quantities.

Of course, this depends a lot on whether we can push down active cases so low.

South Korea seems to be nearly or already there. Of course we do not know whether China tells the complete truth, but even if they lie quite a bit they are also probably nearly or already there. Outbreaks with exponential growth haven't happened there at least, of that we can be relatively certain (because that would be hard to hide).

So, yeah, it's probably possible if we can push effective R a bit below zero for a couple of weeks to get into a position where we can regain control. Maybe. Hopefully.

<comment><USER22> I don't disagree with the plan. It's our only option. But how will we implement it? We have no idea how many active cases their are in the NY area (I assume you mean NYC). And we don't know how many people require hospitalization. So you cannot estimate back from the hospitalized numbers. Plus, that would only tell you how many people we infected a week ago. Except that even that isn't certain because we don't know how long between infection and hospitalization. And all of these numbers depend on what pre-existing conditions (including age) a patient has.

So are we over the worst of it or is it just starting? Should we increase or decrease our lock down right now?

And before you answer, getting it wrong won't be noticed for at least a week (or more if the 1 week incubation figure is wrong, which it is), and in that time any mistake will kill tens of thousands or burn $100bn dollars.

This is what we get for testing senators and NBA stars when we should have been studying populations. China blind sided us with this once by lying about it and covering it up. But we have blind sided ourselves as well by lacking any leadership and ignoring experts. This whole thing as been a huge lesson in how badly western governments are currently running. The slightest hint of a problem and it's every man for himself.

</comment></comment></comment><comment><USER24> Good post discussing Rt (rarely brought up), but a few points are lacking:

* in the face of increasing (and volatile) testing capacity, Rt is being significantly over-estimated by looking at case counts. It's improbable CA still had an Rt above 1 the first week of April with covid deaths linear by the second week. More likely Rt was 1 about a week after the SIP (late march).

* Ignoring "herd immunity" effects of Rt. Some of the "under control" states (NY, Louisiana) have had very high infection rates which in their own end has dropped Rt down. NYC is a strong example of this - with > 20% of the city having been infected, well, Rt will drop dramatically just by so many contacts already having immunity.

* Implying this argues for states locking down now. I agree there is heavy evidence favoring lockdowns weeks ago to have avoided many deaths, but at this point, any lockdown is going to take a week to have an impact on even confirmed cases. The most susceptible populations (essential workers) are the very ones exempted from lockdowns (and less susceptible ones have already voluntarily socially distanced), so there's likely not much gain to be had. Sweden is conveniently our low density control group using mostly voluntary measures (WA State is similar through March 23) and its peak passing a week ago is a sign that Rt drops under 1 faster than you think.

</comment><comment><USER25> I also would like to see this broken down by city because the state highly dilutes the signal when trying to rate the responses of major metropolitan governments.

</comment><comment><USER26> Are there similar calculations for European countries? Several countries are easing lockdowns based on the idea that the transmission rate has become manageable. I'd love to see how those numbers stack up to US states.

EDIT: Turns out there is... [https://epiforecasts.io/covid/posts/global/](https://epiforecasts.io/covid/posts/global/)

</comment><comment><USER27> I would be really impressed if it were possible to control for the differences that would occur if one state did a very good job tracing potential cases and _then_ testing them (Massachusetts), versus more ordinary blanket efforts elsewhere.

<comment><USER10> I think we will see the overall work when we look at deaths. It is lagging though.

<comment><USER28> The author needs to re-run the analysis using only deaths, or deaths plus hospitalizations (if those figures are available). I fear that all the model reveals as currently written is how well states are getting a handle on their testing, and paradoxically showing states with better testing as worse off because of the growing number of cases.

</comment></comment></comment><comment><USER29> Your second rank ordering chart, the one by the upper bound of Rt, is deceptive and misleading. North Dakota and Arkansas only have a high upper bound because there are so few cases that the uncertainty is high. That chart and any analysis associated with it are completely without merit.

</comment><comment><USER30> Now ask yourself - do you believe that Louisiana has a more stringent, more accepted and more enforced lockdown than say Massachusetts or Pennsylvania? Yet the epidemic seems to slow more drastically in Lousiana than most other states.

Why is that? Investigating that seems to be more fruitful avenue.

Perhaps there is more validity to the initial theory that the virus is a lot less virulent in high humidity and heat. Many more observations seem to back that. Low number of cases in India or Brazil etc.

<comment><USER31> You're cherry-picking. Louisiana is on the low side, but Alabama, Texas, and Georgia are on the high side.

In noisy and unreliable data it's _always_ possible to pick a single data point that supports any desired conclusion. I'd say that all we can conclude is that everybody should stay vigilant.

<comment><USER32> >In noisy and unreliable data it's always possible to pick a single data point that supports any desired conclusion.

Reminds me of the Principle of Propaganda - You can prove anything if you ignore enough facts or data.

</comment></comment></comment><comment><USER33> So cool to see Kevin doing a deep dive on this crisis with the goal of making an impact. It says a lot about his character that he chooses to spend his time doing this.

<comment><USER34> But also I think that it's tiresome for people without background in study of disease (or long experience like Bill Gates) to get involved. Not only this but it can be dangerous with any area like diseases.

This is the same as problem that you see in the White House of America with this son-in-law of president. Systrom has no experience with infectious diseases or study of diseases (or I did not hear about it yet and he does not say about it), and human society needs very much people who are experts right now.

Maybe Systrom could also use tech to help as this is his area of expertise.

</comment></comment><comment><USER35> Totally off-topic: interesting, it's the first usage of the default 2020 wordpress theme I see in the wild.

</comment><comment><USER36> Does having a popular mobile app make people scientists? This elitist shit is annoying. I would much rather see him write up something about how he is distributing his wealth to help with the coronavirus rather than this stuff.

<comment><USER37> He sold Instagram years ago, so it's no longer his. Also, instead of blindly launching an ad hominem attack against him, judge the article on its contents. Of course, having success in one area doesn't mean he is an expert in another, but it doesn't exclude the possibility as well.

</comment></comment><comment><USER38> Edit: Sorry, I made a mistake.

Trying to measure Rt seems to be a good idea.

However it's not clear how the proposal calculates Rt. No formulas and no references to raw data. So I am afraid this is not very useful.

This said, I could imagine that an approach which takes the number of confirmed cases and the total number of tests could work to estimate Rt. The idea is that while the number of cases is not a good estimator we can try to get a better estimator using the number of tests as well. It's a bit tricky, however, because a low number of tests could mean that only the probable cases are tested which skews the numbers.

<comment><USER39> Did you miss the link to the notebook containing all the code and calculations?

<comment><USER38> Yes, I goofed up. Sorry.</comment></comment></comment>


Wikipedia and Dark Side Editing - <AUTHOR> https://www.gwern.net/Wikipedia-and-Dark-Side-Editing <comment><USER2> I started writing a comment in favor of the author's main point, and then I convinced myself against it.

The author considers "opting out of Wikipedia" a "nuclear option" that "often won't work". But there's nothing wrong with writing your own book/wiki about a specific topic.

Knowing that "the character of Kaworu Nagisa was originally a cat" is not going to increase your understanding of Evangelion. Would I wish for that specific piece of knowledge to be lost forever? No. But it would be much better served in a book/wiki dedicated to the minutiae of the series than in a general encyclopedia: it would be better targeted to a specific audience, and it wouldn't have the citation problems the author mentions here.

As a side note, the piece was written in 2013. Evangelion wikis are alive as well.

</comment><comment><USER3> > But the overwhelming majority of Eva material--interviews, articles, and > books, to say nothing of the usual merchandise--will never appear in > English.

Not a problem per se, since Wiki allows referencing non-English sources. Of course, an English-language source is preferred whenever possible.</comment>


Easy i18n in Angular Apps - <AUTHOR> http://pascalprecht.github.io/angular-translate/

<comment><USER2> What's obsession with those TITLE, PARAGRAPH_ABOUT_FOO, SECOND_SENTENSE_FROM_TO_RIGHT_WHERE_THEY_KILLED_THE_RABBIT stupidity?

translate("This is a paragraph.")

PS and in the real world, please don't translate language names with each language change

<comment><USER3> Off hand I can think of a few reasons for using the mapping object as they did in angular-translate:

\- It may make sense to translate the same English sentence to something slightly different in different contexts in German.

\- JSLint/JSHint should be able to check for usages on an identifier like TITLE, PARAGRAPH_ABOUT_FOO, etc. easier than comparing strings (you'd have to build something custom for that).

\- Brevity. If you have the same long string of text written twice it may be useful to have a shorter identifier for it.

I agree with not translating language names though, in the real world that doesn't really make sense.

<comment><USER2> > _It may make sense to translate the same English sentence to something > slightly different in different contexts in German_

Then obviously you add context. It all has been solved for xx years [http://www.gnu.org/software/gettext/manual/html_node/Context...](http://www.gnu.org/software/gettext/manual/html_node/Contexts.html#Contexts)

> _JSLint /JSHint should be able to check for usages on an identifier like > TITLE, PARAGRAPH_ABOUT_FOO, etc. easier than comparing strings (you'd have > to build something custom for that)_

That's not difficult if you absolutely need it

> Brevity. If you have the same long string of text written twice it may be > useful to have a shorter identifier for it

You don't need them twice. Add attribute telling: this text can be translated. That's all.

</comment></comment></comment><comment><USER4> angular-translate is really a pleasure to use. I used it in two separate smallish apps and wouldn't hesitate to do it again. Very compact, easy API and great docs.</comment>


The Girl Effect - <AUTHOR> http://www.women2.com/vc-ben-horowitz-the-future-of-humankind-is-dependent-on-technovation-girls-the-girl-effect/

<comment><USER2> Apologies for the tone in advance, but I have to call complete and utter bullshit on this one.

It was a man who convinced my father to buy me a computer when I was but 8 years old. It was he who got me interested in all things tech.

It was a 16 year old boy who taught the 11 year old me how to write my first BASIC program, and under whose tutelage, I wrote my first game.

I attended a boarding school run by an institution which was started by a man (Edmund Rice) who devoted his life to education.

It was a man whose book inspired me to take up science at senior secondary school (The man was Carl Sagan, and the book in question was Cosmos).

While I do not oppose creating a world where everyone can aspire to his or her goals without significant obstacles, I cannot abide such denigration of a gender, be it male, female or otherwise.

</comment><comment><USER3> Does anyone have any idea where he gets the figures that girls educate four people and boys educate no one? Is there any real substance to that claim?</comment>


When Will Microsoft Get a Clue? - <AUTHOR> http://www.fastcompany.com/blog/chris-dannen/techwatch/when-will-microsoft-get-clue <comment><USER2> Sounds pretty shitty to me. Doesn't everyone buy their music on iTunes?</comment>


I made a beer website, brooski.net, check it out. Let me know what you think. - <AUTHOR> http://brooski.net

<comment><USER2> Neat. One thing I might have done differently was to not make ratings on a 0-100 scale. We learned this the hard way at Cork'd; what's the real difference between 85 or 86? To the untrained palette, not much.

Also, curious how you're dealing with normalization of beer names; also an interesting challenge.

Good stuff overall; the site is fast and seems like it's got the right mix of features to make it fun to use. Best of luck!

<comment><USER3> +1 for the rating advice. Make the beer ratings 0 to 10.

<comment><USER4> look how facebook or youtube are doing it... like or dislike. should be that easy. also I don't recall a situation where I thought: "well, maybe I like this beer"

</comment></comment></comment><comment><USER5> You might want to consider getting involved with the Open Beer Database (for the beer listings themselves) rather than become yet another island of beer information on the web.

"The goal of the Open Beer Database is to centralize basic beer and brewery information. We respect and enjoy all of the beer related communities that have sprung up across the internet, however, our main concern is the non-open nature of these community's databases. The continual duplication of this data will eventually lead to incomplete and/or out-of-date sources for basic beer and brewery data."

<http://openbeerdb.com/>

I'm a BeerAdvocate.com user but I do think that there is room for a more social rating/sharing site that is more focused on questions like: What are my friends drinking/recommending? What do I like? What is available near me?

edit - I'd recommend looking the "BeerAdvocate Talk & Site Help " board on BeerAdvocate and picking out the suggestions that are related to social sharing of information and ratings. The site is very much focused on long-form reviews and these types of suggestions are often shot down, but the ideas and opinions should be valuable to you.

</comment><comment><USER6> My suggestion would be to take the "Added" activities off the main feed, as it looks a little redundant to say "Someone drank [X beer]", immediately followed by "Someone added [X beer]".

Besides that, on the About page, the word "remember" is misspelled and I would capitalize Miller/Coors/Bud to make it look more professional.

But I like the idea!

</comment><comment><USER3> Nice work. One thing that confused me: I register for an account and sign in, but when I try to comment on a beer, it asks me to login through Disqus or Facebook or OpenID, etc. Having 2 different authentication schemes in the same site is jarring.

</comment><comment><USER7> Needs a better color scheme and possible a picture of some beer.

What is a drink when it comes to beer? I am not american, so that might be a language issue.

</comment><comment><AUTHOR> You can now add pics too.</comment>


John Mayer: "Manage the Temptation to Publish Yourself" - <AUTHOR> http://www.berklee-blogs.com/2011/07/john-mayer-2011-clinic-manage-the-temptation-to-publish-yourself/ <comment><USER2> I think a few of the commenters are misinterpreting the message he was attempting to convey (or maybe I am!).

What I took from that article was that, as a musician, spending too much time making youtube videos - posting stuff up on myspace music, keeping up with soundcloud track comments, &c... (those are my own examples of what I _think_ he meant by online social media) harms your ability to focus, to create, and to ultimately cultivate a fan base that has a deeper appreciation for your music than the fleeting thousands that skip impatiently through a youtube video then go onto the next artist's video.

I fully agree, publishing yourself using social media is crucial - but so is doing live tours, shows, showing up to a random small beach party and playing some music (that will win you some fans), and spending time on your own creating...

That's what I got out of it, at least, please correct me if I misinterpreted.

<comment><USER3> I had a similar thought when I saw the Ok Go treadmill video about the same time as I was reading Keith Richard's memoirs. If the Stones had spent as much time obsessing over choreography of their Youtube videos in their early days as they did just playing gigs, would they have ended up as the Stones? Will we ever see a pop band again with a repertoire that deep?

Then again I guess the Richards himself had time to nurse a heroine addiction and do some other extraneous stuff at the height of his creative powers. So maybe it is possible to be a prolific music composer and obsessed with social media. Just avoid the heroine.

<comment><USER4> FWIW, heroin is the drug, heroine is a lady hero.

<comment><USER5> I think he had a heroine addiction as well. Or wait, maybe that was Bill Wyman who was addicted.

</comment></comment></comment></comment><comment><USER6> This guy is SO out of touch with independent music it's not even funny. How many people know amazing, talented bands that get no where? Good music within itself does not promote itself. You have to advertise, whether it's engaging in fans online or offline, or making friends with other musicians to split the bill. Do not have a backup plan? Is he insane? For every insanely lucky guy like him there's thousands of other guys working off their blood, sweat, and tears only to end up extremely poor or with nothing. What he's saying is like every startup is going to be Facebook or Google... I'm sorry that's not going to happen. But at least even in a startup you tend to make decent money, not near poverty wages like touring musicians have to endure for years before any "break" ...

<comment><USER7> To be fair, John Mayer _started_ as an independent. His first album was self- published. He did a tour where he drove from gig to gig in his own car. Even after getting a contract, he played unglamorous gigs second-fiddle to guys like Glen Phillips (the lead singer of Toad the Wet Sprocket) at dinky clubs in Seattle.

Sure, he's hit the mainstream, but the guy knows something about being independent.

<comment><AUTHOR> I second this. I haven't been too big of a fan of his latest work, but I bought my first John Mayer album physically from John Mayer. To say he doesn't understand independents isn't true.

<comment><USER7> Me too. I have a signed copy of his second album that I bought from him directly, by handing him cash, in a club. That's about as independent as it gets.

</comment></comment></comment></comment><comment><USER8> I, like Ixiaus, also took something different away from the article than most of the comments. I have several smart, talented friends who indulge in social media to a gross extent. They are the kind of people, like Mayer, that are at the point where they are asking questions like, "Is this a good blog? Is this a good tweet? Which used to be is this a good song title? Is this a good bridge?" but for their respective fields/talents. I certainly wish they would invest less time into those questions and more time into their craft/themselves.

</comment><comment><USER9> I couldn't disagree more. Publishing and promotion is so easy and cheap these days it should be the default option of every musician, book writer, etc. to self-publish.

Sure, it'll take some work to maintain a blog (or tumblr), use twitter, reply to emails, use facebook/g+, etc. But if you do it right it'll just be a very natural way to keep in touch with your fans and it'll be a rewarding experience on its own. The benefits are too huge to ignore. For one, complete control of your music and your career. For another, vastly increased profit margins enabling you to avoid having a day job at a far, far lower threshold than if you had used a more traditional publishing model. And ultimately a much more significant and meaningful connection between you and your fan-base, making your "work" all the more rewarding for both you and them.

<comment><USER10> What is the point of publishing if you have nothing interesting (yet) to say? He is talking about developing a voice before you scream.

<comment><USER9> That's a misleading dichotomy. All artists develop and mature over time, even at the start. You don't need to be fully developed to nevertheless have meaningful work and artistic talent that others would appreciate. Look at xkcd, for example, imagine if randall munroe had said "maybe I should wait until I have something more meaningful to say, and until I have better drawing skills".

Sure, maybe you don't quit your day job if you're still incubating a nascent talent, but you don't have to hide it away from the public. Indeed, one of the best ways to improve artistic skills is to merely exercise them a lot, and to get your work out in front of thoughtful critics and fans. Don't force artistic development into a waterfall process, it can be iterative and agile.

<comment><USER10> I think his general premise here is that the ease and ability with which one can generate and publish content instantaneously -replete with instant feedback loops- sets up bad mental habits and subroutines that begin to inhibit the ability for John Mayer to make great content.

There is a gestational element to most great creative work that cannot be denied. I think to protest this too much this suggests an infatuation with tools and process, not the product.

</comment></comment></comment></comment><comment><USER11> I really liked his thoughts on this. While I understand social media is a powerful form of promotion, its not the be all end all, there is more too it, especially when you do something outside of social media that you are promoting. The part about going for the little applause was great, you get hooked on each tweet, each video, each blog post and its performance in regards to audience reaction, instead of the entire body of work you've created. It has the ability to chop your "art" into micro bits, because it has got to fit inside the constraints of social media. In a way you become your product, it sounds like he stopped being a musician and was selling John Mayer on Twitter, it wasn't about the music anymore.

Which is kind of hard to stop doing, because its what the majority of the internet audience wants. Jeri Ellsworth made an interesting statement about the electronics/hacks she creates, the audience likes the simple ones, and don't care much about the much more technical and in depth videos she creates. The audience to some effect seems to be only interested in the short and simple. Which I would imagine, if John Mayer spent all his social media time, talking about music, and music theory, and the kind of things you would think someone would want to know from a talented artist, I doubt he'd have the followers. Where as, if he sticks to the one liners, and small talk, its vastly more popular.

As for the mental changes is causes, I fully understand this. I've read a bit about how the human brain changes according to how it is use. And also, when I was still blogging, I noticed an interesting change in myself. I started consuming the world in a "how can I blog this" sort of way. Anything even remotely interesting that happened, instead of me enjoying the life experience, quickly turned to, how can I phrase this on my blog? The short time I was on Twitter, I found I spent more mental capacity, sorting through my thoughts to Twitterize them, than on the actual thoughts. If you're famous for social media, that is great. If your a musician, author, electrical engineer, etc, it has the side effect of changing how you think, and then how you create, if you focus on it too much.

</comment><comment><USER12> You have to remember the audience is a college of music. In a venue like this, the primary question is not how to become a commercial success, it's how to become better at your craft as a musician.

Mayer is also speaking from the unique perspective of being a serious commercial success, wanting to nurture it, but still having the "burden" of creating music to (his) high standards. I don't think most normal people need to create such a black and white line between posting on Twitter and writing a song, but Mayer needs to, and it's at least somewhat interesting to hear his reasons.

</comment><comment><USER13> I get what he's saying about it destroying his creative process, but he's clearly forgotten what it's like not to be popular. Advertising yourself is the ONLY way to get fans. They do not just come to you automatically.

<comment><USER14> Really? I don't think for musicians that "Advertising yourself is the ONLY way to get fans". That's the fallacy of modern celebrity. For musicians the music should attract fans, not the musician talking about himself and his work.

<comment><USER13> He specifically talks against advertising your music on social networks, and not just your talking about it.

Getting a gig somewhere local without having fans isn't much fun. Anything you can do to jumpstart that is probably a good thing.

It's almost like telling programmers not to put code on Github and share it because it takes time away from the coding itself.

The truth is, sharing the music is beneficial in itself, and it's not really like you can only do 1 or the other.

He had a Twitter addiction and took it to extremes. It was ruining him. That's a lot different than simply using it to create a fanbase or interact with your fans.

</comment></comment></comment><comment><USER15> Some context: Jennifer Aniston dumped him over his Twitter addiction:

[http://www.telegraph.co.uk/technology/twitter/5038203/Jennif...](http://www.telegraph.co.uk/technology/twitter/5038203/Jennifer- Aniston-ended-relationship-with-John-Mayer-because-of-his-Twitter- obsession.html)

</comment><comment><USER16> _Mayer shared that he found himself asking himself questions like "Is this a good blog? Is this a good tweet? Which used to be is this a good song title? Is this a good bridge?_

As much as I disdain this guy's style sometimes, he makes some excellent points. I take issue with this particular quote though. Perhaps his true calling is to be both a musician and commentator. There's nothing inherently wrong with letting oneself evolve and change focus. It's not like music is a dying art form or something</comment>


Tesla Model Y - <AUTHOR> http://www.autocar.co.uk/car-news/new-cars/tesla-model-y-lead-ambitious-range-expansion-plans <comment><USER2> Everytime there is some negative news, out comes a random Tesla announcement.

How about some Model 3 details, as opposed to pipe dreams that may or may not appear on some random, future model?</comment>


Why Not Python - the GIL hinders concurrency - <AUTHOR> http://www.chrisstucchio.com/blog/2013/why_not_python.html <comment><USER2> NumPy developer here. First, I agree that the "the GIL is not an issue" is an annoying meme. It can be an issue, and when it is, it is annoying as it complicates some architectures. I don't think those architectures are as common as people usually think.

A few remarks: I would qualify the GIL as a tradeoff rather than a mistake. You loose CPU-bound parallelism with threading, but you gain easy to write C extensions (well, relatively speaking). I think this point is critical to the existence of something like NumPy (which is unrivalled in general PL AFAIK).

If you need to share a lot of data (a big numpy array), then you can use mmap arrays, there is no serializing/desarializing cost, and that's efficient. See for example some presentations by O. Grisel from scikits-learn ([https://www.youtube.com/user/EnthoughtMedia/](https://www.youtube.com/user/EnthoughtMedia/) \-- disclaimer, I work for Enthought, but all those video happened at scipy 2013).

IMO, the only significant niche where this is an issue is reaching peak performance on a single machine, but python is not really appropriate anyway -- that's where you need fortran/C/C++, and even scala/java/haskell are quite unheard off there.

<comment><USER3> > You loose CPU-bound parallelism with threading, but you gain easy to write C > extensions

I'm extremely interested in your thoughts on this subject as a NumPy developer. To what extent do you think C interop designs like Python CFFI (which mimics LuaJIT's FFI) will make this an unimpressive accomplishment?

As a former heavy user of Python for scientific programming (for easy access to C libraries) I have moved everything I do to LuaJIT and C. The ease of writing a LuaJIT FFI (or Python CFFI binding) is at least an order of magnitude greater than that of writing PyObject* style bindings.

Do you think it's possible (or likely) that a mature CFFI will make this tradeoff you describe seem no-longer-positive?

Added in Edit: I should point out that multicore patterns in Lua are already very different from those in Python, e.g. running a Lua VM per core and communicating through shared memory. So I realize that the existence of CFFI won't map one-to-one onto changes in Python multicore design, at least in the short term.

<comment><USER2> For interfacing with C, I think cython is better than cffi. Well, I don't have much experience with cffi, but I used to use ctypes, and cython is much better. Cffi may be nicer than ctypes, but not enough to make me switch (I may be missing something).

Unfortunately, I think that for python, it is too late to have a better C API: there is so much legacy that depends on python C API details that moving away from that would take almost as much manpower as rewriting it to a different language.

Look at pypy: even though it is 5x times faster on average than python, I don't see people flying from cpython to pypy. 5x speedup is more than what you can hope on todays' CPU with near perfect scaling with # cores. This tells me that no-GIL python, to be successful, would need to have a much lower barrier to entry than pypy. Maybe the future is closer to having hooks to 'escape' python for the numerical stuff: numba and the numerous similar projects jitting a subset of python, or interoperating with some other language more amenable to optimization (e.g. Julia: [http://www.youtube.com/watch?v=Eb8CMuNKdJ0](http://www.youtube.com/watch?v=Eb8CMuNKdJ0))

I am quite interested in Lua-like design, though: one of my pet project around numpy would be to refactor it internally to use something closer to how lua works, exposing numpy to cpython only at the outer layer.

<comment><USER3> > For interfacing with C, I think cython is better than cffi.

I think we'll have to agree to disagree on this. To quote the CFFI project description [0]: the goal is to be able to seamlessly call C from Python without having to learn a third language/API. Cython, Ctypes, and CPython's C API all fail to meet this criterion, and in my opinion do so by a large margin. Let's also not forget the fact that Cython requires a compilation step not only for the bound C code but also for the portion of the "Python" code that interfaces with it.

Compare that (or writing CPython C API manipulation code) with the following:

      >>> from cffi import FFI
      >>> ffi = FFI()
      >>> ffi.cdef("int printf(const char *format, ...);")
      >>> C = ffi.dlopen(None)                     # loads the entire C namespace
      >>> arg = ffi.new("char[]", "world")         # char arg[] = "world";
      >>> C.printf("hi there, %s!\n", arg)         # call printf
      hi there, world!

For me CFFI wins without question. Of course, it's even simpler in LuaJIT (but the CFFI people are getting closer every day):

      local ffi = require "ffi"
      ffi.cdef [[ int printf(const char*, ...); ]]
      ffi.C.printf("hi there, %s!\n", "world")

Now, it would be misleading for me not to point out the limitations of an approach like CFFI/LJ-FFI vs CPython API: you may have to contort yourself to make C calls that safely manipulate the scripting language VM (e.g. instantiate new Python objects from within a C FFI call). In my experience this has not been a serious problem because it's easy to write struct-to- object mappers in idiomatic Python or Lua rather than having to muck about with PyObject* and friends. Of course, in LuaJIT it's even easier with FFI metatypes (which let you attach dispatch tables to FFI native types, so your structs can behave just like Lua tables with method dispatch, inheritance, etc.).

Anyway, leaving personal taste aside for a moment, let me rephrase the original question that sparked my interest in your opinion: in a world with a mature CFFI option providing the above binding capabilities do you think that easy C extensions are still a worthwhile tradeoff for the GIL? (FWIW I find the point to be quite compelling for _why Python had a GIL to begin with in the late 90s_ , but I'm suspecting it's less of an intrinsic design tradeoff and more historical cruft as the PyPy and CFFI people roll out their work).

[0] [http://cffi.readthedocs.org/en/release-0.6/](http://cffi.readthedocs.org/en/release-0.6/)

<comment><USER2> I don't think the points you highlighted above matter fundamentally: they don't handle memory management for you, and that's the difficulty, especially at the language boundary. What is interesting for me in Lua is the stack-based argument passing and the GC: it avoids leaking the reference counting and the binary representation of objects (I think, I am rather clueless when it comes to Lua).

Wrapping things like crypto, file format and co is relatively trivial, because not much crosses the language barrier. Now, when you need to handle non native object life cycle, that's another matter, and I don't see how cffi makes it any easier than cython

<comment><USER3> This thread has probably gone too long, and I thank you for your time and comments so far.

I will close by saying that, while I agree with you that CFFI doesn't make the problem of writing memory-safe C extensions to the Python VM any easier, I also believe that almost no one (aside from people like Numpy developers) actually _need_ to write a Python extension. They just write extensions (or use Cython, or Ctypes) because that's "the way" to call C from Python.

In my personal experience 90+% of the Python+C problems in the world are just about calling an existing piece of C code and maybe mapping some structs or arrays into Python types; these workflows rarely involve instantiating complex objects on the Python side. For my work habits neither CPython, Ctypes, nor Cython are optimized for this extremely common use case. CFFI solves these problems for me better than any of those alternatives, and if someone offered me a Python 4.0 with CFFI and no GIL at the cost of harder-to-use extension mechanisms I'd jump on it; I suspect that a significant portion of the people who do Python/C interop work would feel similarly. I also do realize that such a change could make projects like Numpy harder to build.

</comment></comment></comment></comment></comment></comment><comment><USER4> All of these problems can be addressed using inter-process shared memory. Shared memory support is built into multiprocessing [1].

Now I agree it might not be convenient, but that's a matter of libraries. This post would be much more constructive if it was speculation on what such a library could look like, instead of pretending that concurrency "doesn't work" in single-threaded Unix processes.

1\. [http://docs.python.org/2/library/multiprocessing.html#module...](http://docs.python.org/2/library/multiprocessing.html#module- multiprocessing.sharedctypes)

<comment><USER5> Let me first say that I agree with you.

However, I think that the threaded model is in general easier to work with, rather than shared memory. That said, this might be offset by python itself being easier to work with than another language.

Also, separating work into multiple processes gives in general better security - one crashing worker won't impact the others. Then again, you won't take advantage of hyper-threading features of the CPU.

In the end, everybody needs to consider their personal needs and make a tradeoff.

<comment><USER6> Can you please explain why using processes instead of threads means you can't take advantage of hyper-threading? It may seem obvious from the name "hyper- threading", but I want to make sure that this inference is accurate.

<comment><USER4> It shouldn't. Processors with hyper-threading enabled expose two virtual processors to the operating system per physical core.

</comment></comment></comment></comment><comment><USER7> Nice post, Chris, and it's definitely a problem that many people are confused about (as the comments in this thread show). People who think the GIL is a problem either have no idea what they're talking about, or they really know what they're talking about. People who don't think the GIL is a problem either have no idea what they're talking about, or they really know what they're talking about. :-p

One of the lesser-appreciated facts about the GIL is that _it is an implementation detail of CPython_. That is, it is entirely possible to implement a Python that does not have process-level globals and C statics. There is no _structural_ reason why a C or C++ implementation of Python _needs_ to have a GIL. It's just a legacy of the implementation that has stayed around for a long, long time.

For instance, Trent Nelson has done some work to show that you can move to thread-local storage for the free lists and whatnot, and get nice multicore scaling, even with the existing CPython codebase[1]. There are still concurrency primitives that the language would need to offer at the C API level to manage the merger of these thread-local objects, but it's a whole lot better than only being able to use a single core in modern days.

Fortunately I mostly get to work in the scientific field with (mostly) data parallel problems.

[1] [https://speakerdeck.com/trent/parallelizing-the-python- inter...](https://speakerdeck.com/trent/parallelizing-the-python-interpreter- an-alternate-approach-to-async)

</comment><comment><USER8>
      But fundamentally, the GIL prevents Python from being used as a systems language.

This is only true when severely restricting the definition of a systems language. The vast majority of command line utilities you'll find in a typical operating system are often primarily I/O-bound or do not have critical performance requirements.

As such, I'd only be willing to agree with the author's statement for a very specific subset of systems programming.

For example, I've worked on a packaging system written in Python for the last five years or so. The package system is primarily I/O-bound the vast majority of the time (waiting on disks or network), and almost all significant performance wins (some as much as 80-90%) have come from algorithmic improvements rather than rewriting portions in C (very little is written in C).

As one of my colleagues is fond of saying (paraphrasing), "doing less work is the best way to go faster".

It also ignores the fact that depending on the problem space involved, there may be readily available solutions that provide excellent performance that don't involve threading (e.g. the multiprocessing module, shared memory, etc.).

</comment><comment><USER9> This feels to me more like a critique of fork() and the unixy process-oriented parallelism model than a critique of Python. Of course, the author mentions this as a caveat, but it makes me wonder if the blame is being laid where it should be (and also whether in some scenarios like this, you should really just build applications the way applications are normally built for a platform, no matter how much you dislike it)

<comment><USER10> Multiprocessing is touted to work around the GIL. I too have run into the wall trying to get big problems solved in Python. And when I went multiprocessing, I ran into limitations in its internals e.g. using `select()` that really surprised me too.

</comment></comment><comment><USER11> He has a valid point. There are certain types of workloads which don't scale well on a single machine when writing pure python code. If your workloads happens to need to get more cpu performance out of a single machine in pure python and isn't easily to parallelize using processes, python's not the best choice. Personally I feel like go is good for this use case because I consider writing my own object lifecycle code a pita. Others will only use multithreaded c code, something I'm not eager to touch.

I do however feel the need to make a couple clarifications:

If you fork a process, you don't necessarily duplicate memory. Yay for COW. Fork twice and you've got fanout with very little extra memory. Threading works fantastically for I/O. C libraries can release the GIL during processing. So if you have to do something computationally expensive, you can let another python thread run.

I rarely feel restricted by python, and when I do I usually find that other aspects make up for it. I find it makes a decent systems programming language (although not as good as C). The os library and the ctypes library give you the capability to do most (all?) system level tasks.

I tend to prefer go over python when library support isn't an issue, but that's more due to it's type system and channel primitives. As someone who's pushed python to it's limits as a systems programming language I'm pretty comfortable dropping down to C isn't _often_ called for, and writing in java/scala isn't ever required.

<comment><USER2> COW semantics of fork is not that useful with python because of reference counting (at least cpython implementation where the reference count lives inside the object memory representation). It may work much better with pypy (which uses a 'real' GC but still has a GIL)

<comment><USER11> Excellent point. Ruby's take on this has been interesting. [http://patshaughnessy.net/2012/3/23/why-you-should-be- excite...](http://patshaughnessy.net/2012/3/23/why-you-should-be-excited- about-garbage-collection-in-ruby-2-0)

I'm not a ruby expert but by understanding is that basically they've moved the refcount field out of the struct and out of the memory page. It would be nice if python did something like this.

[edit] My summary of what ruby does is totally wrong and while this is interesting and applies to memory managment, doesn't necessarily apply to refcounting.

</comment></comment></comment><comment><USER12> GIL doesn't hinder concurrency, Python's threading library is still concurrent. It just isn't parallel.

<comment><USER13> What? The entire definition of being _concurrent_ is doing work in __parallel __.

<comment><USER14> I believe that is a common misconception. Concurrency enables parallelism but it isn't a requirement.

The golang community has lectured about this at length: [http://blog.golang.org/concurrency-is-not- parallelism](http://blog.golang.org/concurrency-is-not-parallelism)

</comment></comment></comment><comment><USER15> This is just my opinion, but it's served me well in the 8 or so years I've done Python development:

Why are you doing compute intensive work in Python? Python is not well optimized for doing compute intensive work. It's made some strides over the years, being able to do limited bytecode optimization and multiprocessing, but it's not a high performance computation language.

However, it is a fantastic glue language. Write your compute intensive portions in C, and use Python to glue the portions of C together. You can even do your thread generation in Python, release the GIL during your C execution, and you don't even have to worry about the complicated process of spinning up threads in your C.

In other words, use the language like it was designed to be used.

<comment><USER16> > _Write your compute intensive portions in C_

This advice is so often given by people, that I suspect that most people don't know what they are talking about, because in fact most people never do that, being unable to drop to C. I dare you to show me a piece of " _compute intensive_ " portion that you optimized with C.

I've worked with Python for about 3 years. After struggling with it to stretch the boundaries of what it could do, I finally gave up in frustration and the final solution was to find a less limiting environment. Worked great thus far and I don't miss anything about Python.

<comment><USER17> We do this in robotics (research) all the time, eg. for image and pointcloud processing. I've implemented algorithms in plain python, numpy, and then OpenCV / PCL. When a compute-intensive task starts becoming painful, it's profiled, improved, and/or eventually forked into a C project with python- friendly wrappers.

I acknowledge this is very anecdotal... but my (previous) labs observed a tradeoff in developer productivity in Python vs C, and deemed Python worthwhile (avoiding classic premature optimization and whatnot).

</comment></comment></comment><comment><USER18> Timely and related talk given today at EuroPython conference about concurrency in Python and why (and when) GIL doesn't matter: [http://www.youtube.com/watch?v=b9vTUZYmtiE](http://www.youtube.com/watch?v=b9vTUZYmtiE)

</comment><comment><USER19> > Memory duplication has a relatively simple solution, namely using external > cache such as redis. But the thundering herds problem remains. At time t=0, > each process receives a request for f(new input). Each process looks in the > cache, finds it empty, and begins computing f(new input). As a result every > single process is blocked.

this is incorrect. The processes coordinate on a lock held in redis itself.

This solution is available right now using the Redis backend in dogpile.cache (of which I am the author): [https://dogpilecache.readthedocs.org/en/latest/usage.html](https://dogpilecache.readthedocs.org/en/latest/usage.html)

<comment><USER20> It's also not too hard to DIY:

[http://www.dr-josiah.com/2012/01/creating-lock-with- redis.ht...](http://www.dr-josiah.com/2012/01/creating-lock-with-redis.html)

[https://github.com/njbooher/boglab_tools/blob/dece35f13a8fcb...](https://github.com/njbooher/boglab_tools/blob/dece35f13a8fcb9cb5b7eefdee2d6f9916350918/entrez_cache.py#L105)

When multiple processes ask this for the same file one of them downloads it and the others wait for it to finish.

</comment></comment><comment><USER21> no mention of gevent, celery, etc. As someone who runs thousands of concurrent tasks in a mix of process/gevent (one UNIX process for each 100 greenlets across 48 cores on two boxes), I find the OP's toliling rather misguided.

<comment><USER22> The author's use case is clearly very different from yours. He is talking about CPU-bound processes which need to share a large amount of memory with each other. In this case, multiprocessing and message passing is not really the best fit. Multithreading or shared memory results in far less CPU usage and memory duplication.

</comment></comment><comment><USER23> _The standard Python workaround to the GIL is multiprocessing. Multiprocessing is basically a library which spins up a distributed system running locally - it forks your process, and runs workers in the forks. The parent process then communicates with the child processes via unix pipes, TCP, or some such method, allowing multiple cores to be used._

Multiprocessing is _the way_ to do parallelism. Deviating from that should be an exception -- for example, shared memory maps could be used to transfer select data objects instantaneously between the processes instead of serializing/deserializing over a pipe, and only those while still retaining separate process images. Threads were practically invented as a compensation for systems with heavy process image overhead.

I think Python is very Unix in this regard. And that's not a bad thing per se. Unix and Linux can do multiprocessing very efficiently.

<comment><USER24> Orly? If your language is handicapped maybe. In normal languages you have no problems with threads, and the assertion that multiprocessing is the way makes no sense.

<comment><USER25> It's basically impossible to reason about the correctness of any nontrivial multithreaded program.

<comment><USER26> This is true of all programs not written in Z notation...

</comment></comment></comment></comment><comment><USER27> s/Python/Ruby. Title still works.

</comment><comment><USER28> How can the GIL, which is restricted to one process, hinder concurrency? It can only impact 1 single form of concurrency, threading.

Why use threads?

Noone does parallel compute on CPUs these days, not since GPGPUs rocked up almost 5 years ago (and we often use python as the host language, thanks pyCuda!)

Parallel IO then? Well, except that async IO is often far more resource efficient (at the cost of complexity though).

Threading is dead(-ish) because its hard to write, hard to test and expensive to get right.

Concurrency in python is very much alive though.

<comment><USER13> > Noone does parallel compute on CPUs these days, not since GPGPUs rocked up > almost 5 years ago

I want to live in your world where all you are processing is vectors and FFTs in parallel on GPUs and not doing real work (accessing databases, processing data from sockets, etc).

Threading is not dead. It's only crippled in python so everyone wants to invent ways of saying it is dead.

Threading being hard to write is also a fallacy. I use thread backed dispatch queues which make concurrency simple in my language of choice right now. Threading like that is easy thanks to closures and a good design patterns. My apps are entirely async and run heavily parallel and it's easy to maintain and write using that.

<comment><USER28> Accessing databases, processing data from sockets, are not CPU bound activities? I believe you've misread my post.

For all your IO cases, and all your cases are IO, would you, and future maintainers of your code, not be better served with simpler abstractions which permit scaling past a single host?

<comment><USER13> I wasn't referring to the IO bound side of it but the general work involved with everyday generic work that was not something that a GPU can do very well. It's silly to say the answer to doing parallel is to throw it on the GPU.

But referring to the IO side debate, the current design of many of the libraries that you call in the C world are often inheirtly blocking. 'gethostname' for example is a blocking call. There is no async version of it. To use them without contention on your single threaded application, you have to call them from worker threads.

The common pattern is to spin up a thread to call it and do work on it. It's easier often to have your workers be thread bound like that to simplify your code and only lock shared resources when you need them. I can also make a massively async version of all my code that handles everything using async methods and in many cases this is better but it's harder to write and not always an option. Something I have to deal with daily because I run into the C10K problem all the time at work ([http://en.wikipedia.org/wiki/C10k_problem](http://en.wikipedia.org/wiki/C10k_problem)).

Even in the async model though I still want to be running code in parallel and I would still rather build that model up with thread powering it and not multiple processes and shared memory.

<comment><USER28> A GPU, as you know, doesn't exist in isolation. It sits on a multicore host. The load of input data and the writeback of results does not occur from the GPU as I suspect you know. Maybe in future with unified memory this will be possible but not on current devices.

The actual computation, the bit that was previously multi threaded (or more commonly, multi process) on a CPU, now lives on a GPU. I'm not sure what's silly? The compute bound workload, is now done on the GPU. The IO workload is still done on the CPU, in an inherently single threaded fashion. Even when the multi process computation was done on the CPU, load and store operations were still single threaded. This stands to reason since there is no advantage in splitting 500 concurrent hosts connections into 500* CPU cores connections to hit a central data repository with...

I can't think of any code off the top of my head that calls gethostbyname repeatedly. Maybe a network server of some description which is doing reverse lookups to allow for logging purposes? Although that seems inefficient, I can't think of a real time use case for the host name when you're already in possession of the IP, I can only think of logging / reporting uses cases which would be better served doing the lookup after the fact / offline.

If that's a valid example of what you're suggesting, then would the existing threaded code not be more efficiently implemented asynchronously? There's a finite limit to the number of threads you can create and schedule for these blocking calls, at some point you will have to introduce an async tactic. At that point, why not drop the threading altogether?

You say you would rather build a model on top of threads. Why? Does it make your testing simpler? Does it reduce the time for new starts to get up to speed with your code? Does it reduce the SLOC count? Is it simpler to reason about?

I hope you would agree, in all these cases and many more, threading is at a significant disadvantage. I stand by the assertion that its dead(-ish).

The ish qualifier comes from another case we've not discussed, yet!

</comment></comment></comment></comment></comment><comment><USER29> > The implementation would also likely be considerably more complicated than > the 160 linues of code that the Spray Cache uses.

Not likely, using Twisted deferreds and a sane cache-wrapper with herd awareness -- you probably want this regardless of long-running cachables.

Of course, Python 3.2 also has a futures [thread or process] builtin, if that's your thing.

10-20 lines of code.

</comment><comment><USER30> I don't get how having multiple copies of a quote hanging around in memory is a big problem. It's probably less than 100 bytes total (symbol + side + price) and probably significantly smaller than the size of the state associated with the "statistics process."

<comment><USER31> An individual quote isn't a problem, but they don't tend to come one at a time. For realtime processing, serialization costs are usually the biggest issue, as is cache locality.

On the other hand, for batch processing, memory can be a big issue (CPU less so).

</comment></comment><comment><USER32> if you need performance, just don't use Python. You'll be better off using C/C++.

<comment><USER33> Cython is also definitely worth looking at, if you want to write Python code and you need parts of it to be fast.

[http://cython.org/](http://cython.org/)

<comment><USER32> If it's a small part you want to optimize it works great. But if all your project is performance sensitive it becomes a big hack that's hard to maintain.

</comment></comment></comment><comment><USER26> Regardless of other arguments, it basically means that threads in python become only a way of thinking about a problem rather than a way yo utilise hardware fully. It's a shame.

</comment><comment><USER34> The guys at CCP (the makers of EVE Online) seem to be doing just fine with parallelising stuff in Python.

<comment><USER13> No one said you can't be parallel in python. The problem is that you can't use simple threads and must resort to separate processes, shared memory, and IPC to shard out your work that way.

CCP uses twisted which manages to help you with yielding when doing async IO keeping as much work off the GIL when you are waiting on data and sockets and builds in cooperative multitasking concepts to let you yield to other work, but it's internally not multithreaded or multiprocessing out of the box. You still have scale up worker processes in some cases (usually one per CPU you have) to really make it effective.

<comment><USER15> CCP uses stackless, which provides low overhead co-processes, not twisted. Just a clarification.

<comment><USER13> CCP uses twisted on stackless to be 100% clear.

</comment></comment></comment></comment><comment><USER35> The author appears to not be aware of where the big leagues are. Additionally, as is becoming a recurring theme here, he doesn't know about PyPy nor Twisted. This is continually disappointing.

<comment><USER2> If you think twisted is a solution to the problems mentioned in the OP, you haven't understood the problem. Twisted _may_ be a solution to IO bound processes, where you do cooperative parallelism instead of preemptive (aka threads). It is utterly useless for CPU bound processes (e.g.: you want to compute some expensive operation on top of a big numpy array, twisted does not help you with that at all).

<comment><USER25> Twisted would work perfectly for the message passing example given in the article.</comment></comment></comment>


"How Hackers Work" - <AUTHOR> http://computer.howstuffworks.com/hacker.htm <comment><USER2> They were off to a good start...

 _A hacker was a programmer -- someone who hacked out computer code. Hackers were visionaries who could see new ways to use computers, creating programs that no one else could conceive. They were the pioneers of the computer industry, building everything from small applications to operating systems._

Then the rest of it went to hell. They called yahoo mashup developers hackers! Big WTF. At least Stallman and Wozniak were mentioned as famous hackers.

<comment><USER3> "A hacker is someone who hacks"... bah.</comment></comment>


Startup Quote: Steve Case, co-founder, AOL - <AUTHOR> http://startupquote.com/post/2684158876

<comment><AUTHOR> The idea of an entrepreneur is really thinking out of the box and taking risks and stepping up to major challenges.

\- Steve Case (@SteveCase)

<http://startupquote.com/post/2684158876></comment>


Depreciation of Peso Leads to Argentina Installing 2000 Bitcoin ATMs - <AUTHOR> https://beincrypto.com/when-fiat-falls-flat-cryptocurrency-atms-financial-relief-south-america/ <comment><USER2> Title is clickbait and misleading.

Original title: "When Fiat Falls Flat: Cryptocurrency ATMs Offer Financial Relief in South America"

Actual content / message: "Bitcoin ATM companies aim to place nearly 2000 new ATMs in Argentina by this time next year."

<comment><USER3> It also barely agrees with reality. In a recently published survey [1, in Spanish], companies in one of Argentina's largest cities were asked about which IT professionals were required, and how many they would hire in the next 12 months. They answered that 61 blockchain would be needed in the next 12 months, but the number of open positions right now is... 0. So everyone is planning on doing it eventually, but so far no one has actually coughed the money.

[1] [https://webcache.googleusercontent.com/search?q=cache:http:/...](https://webcache.googleusercontent.com/search?q=cache:http://www.lavoz.com.ar/negocios/13-perfiles- laborales-que-busca-industria-de-software- local%3Futm_campaign=SocialShare%26utm_source=whatsapp)

</comment></comment><comment><USER4> Let's solve our economic crisis caused by a terribly volatile currency by replacing it with a different, more anonymous, terribly volatile currency!</comment>


Ask HN: What Do I Do if a Client Refuses to Pay? - <AUTHOR> I contract for a company that took on a client project that has ended up severely over-budget and over-schedule. The short version of events is that the timeline was quoted assuming 3 developers, but one left a few days in and the other lied on her resume and sucked at everything, including straight content entry - leaving just me to do all the work in a timeline meant for 3 devs. No one else was hired until the past 2 weeks.
Understandably, the client has not paid the company I work for since the start of the project. Meanwhile I've been averaging 80 hour weeks trying to get things done in a rush. I took no holidays, a single sick day, and have completely lost my entire summer to this project.
When it was clear we weren't going to make the second deadline, I started capping my invoices at 40 hours per week instead of the actual time I was working on the project so as not to bleed the company. Soon after that, they began skipping pay days (for me only), or only paying partial amounts.
The company now owes me more than $10,000, my wife is rightly on my case about the whole thing, and they don't seem ready or willing to pay any of the outstanding amount.
Two new people have been hired in the past 2 weeks to help finish this project and handle other work that I literally haven't had time to do on this schedule. One of those people has both tech and management experience and has truly helped me fix the final parts of the project into an actionable plan that the three of us can complete quickly.
I've met with my manager and owner of the company on three separate occasions and I've always come out feeling like they listened and the financial situation would be taken care of. Each time I find myself grabbing my own ankles on pay day again.
So I ask HN - what would you do? I need to fix things, preferably without losing that money (we're broke now). <comment><USER2> I'm going to be brutally honest. You have 2 options: lawyer up and sue to get your money or walk away and treat this as a very expensive lesson learned.

Lessons for next time: 1\. _Never_ cap invoices. Instead, cap your work. If there's 80 hours of work, and the company is only willing to pay for 40, guess what? You should only do 40 hours of work (and bill for it).

2\. Don't worry about bleeding the company. What's the worst they can do? Not pay you? Hell, that's what they're doing right now!

3\. If there's a change in the work timeline, the contract should be renegotiated. You mentioned earlier that one of the other developers quit a few days in. That, right there, would have been grounds for a renegotiation. I mean, unless the departure of this dev. was explicitly planned for, it constitutes a fairly radical change in the terms of the contract — you now have to do 50% more work than anticipated.

4\. "I've met with my manager and the owner of the company on three separate occasions and I've always felt like they listened and the financial situation would be taken care of. Each time I find myself grabbing my own ankles on pay day again."

Whoa, this should have been a gigantic, enormous red flag. Words are cheap, especially if you're of the sociopathic type that heads all too many companies. What counts is action. I mean, even if the company is in financial trouble, they still have a contractual obligation to pay you. Out and out missing a payment is unacceptable. At the very least, I'd expect a clear warning in advance and a concrete set of steps detailing how and when the make-up payment will be delivered. And even then, if they consistently miss paying you, fire them.

</comment><comment><USER3> This is meant for designers but inspired me to change my practices: [http://vimeo.com/22053820](http://vimeo.com/22053820)

Its titled: "fuck you, pay me"

The video isn't actionable advice but worth a look.

If I were in your shoes I'd quit immediately once a payment was missed. These situations don't get better in my experience. :(

</comment><comment><USER4> You work for the company, not the client. The fact that they haven't been paid should be irrelevant to whether you are paid or not.

If this company is living hand-to-mouth for payroll, are they going to be around for a while, anyway?

Honestly, capping your invoices and continuing to work without being paid only enables them to be shitheels.

You need to tell them that you will resign unless you're paid, in full, immediately.

I would also consider talking to a lawyer.

(Please don't take any of this as criticism -- I've been in your place before!)

</comment><comment><USER5> Lawyer up. As long as you can prove your case, they owe you. If you have a hard time obtaining supporting evidence, then if you have a good lawyer, he/she should be able to pressure your company into a settlement.

Do not just walk away. That's unfair to you, and the company is just going to do the same thing to the next guy. You may not care about the next guy and his problems, but think about it this way: the company you now hate is going to get more free work. Don't let them.

Do not work for free. $10,000 is not a lesson, it's a crime.

</comment><comment><USER6> If you're not getting paid, then you currently don't have a job, go look for one. You have zero moral obligations to them if they're blatantly not doing their part of the deal.

At the same time, you have a debt to collect - either through negotiation or legal action.

</comment><comment><USER7> I don't really see the problem, you worked for a company, they have to pay you, if they don't just sue them...

I mean, definitely is not your (direct) problem if the client haven't pay yet and is not your fault neither that you missed the deadlines...

I would just be straight with your manager...

</comment><comment><USER8> Lawyer up, baby!

Seriously. Start by paying a lawyer to write a nasty demand for payment note with a hard deadline. This works for me 95% of the time.

</comment><comment><USER9> I'd do:

1) Tell them you're not comfortable enough with your relationship to do any more work until you see the money you're owed. Might also be worth pointing out they'll still owe you for work done and that they'll incur additional expenses searching for someone else to pick up the project.

2) Start looking for other work, this isn't making you any non-theoretical money at the moment.

3) If 1 doesn't work lawyer up or walk away, depending on whether you can afford a lawyer.

4) Never work with them again.

</comment><comment><USER10> Are you actually me? This sounds so familiar, I just had a flashback...

Get a lawyer, like everyone else is saying. The fundamental fault I made was not enforcing my boundaries - this is the purpose of lawyers, to help you do this, but you first have to be aware of it yourself. So if you are not very familiar with contract law, learn the basics and how to analyze a contract etc. Apologies if I'm assuming things, but my assumption based on what you are saying is that you aren't.

I would recommend this to -everyone- in the IT field - there is now far too money sloshing around and every clever asshole out there is hell bent on becoming the next Steve Jobs. And learn a bit about the Loser/Clueless/Sociopath theory of management - I ended up working for a bona fide sociopath, as it turned out...

I too quasi-capped my invoices on this hell of a project - I lowered my rate for certain types of tasks, given they were mostly data entry like stuff, so it seemed logical and at that point, I was psychologically mired in the whole thing. Somehow this website I was hired to build turned into my being the entire IT staff for a enterprise B2B startup. As in everything - sysadmin to QA to CTO.

I finally snapped out of it when a) the CEO lied to a customer about the data we were selling - blatant negotiation in bad faith and the customer was going to sue (and mostly definitely win) and B) it became apparent how they were playing 3 card monte with the invoices and lying to all the salesguys and me about cash flow issues while they paid off the loans their OTHER company had made to fund the startup.

So everyone working for this company quit. It took us 6+ months to get paid back because the CEO's wife then went and started pulling crap, reinterpreting contracts and claiming people were not owed so much money...

I too, like lots of contractors, have a commitment to the job and professionalism that you exhibited - I am certainly not faulting you for that. But you have to realize some clients will consciously take advantage of that - read up on manipulative people and how to deal with them. A good book is "In Sheep's Clothing" by George Simon.

Then I suggest some deep soul searching / psychoanalysis to figure out what your blinds spots were and why - it was all a valuable lesson for me and I suppose it would have happened someday. I can say it was ultimately useful. But 4 years of hell is a long time to waste.

So I hope you can find the silver lining in all this and I wish you luck.

<comment><USER11> +1 on this. Thanks also for the book referral, seems like an interesting book.

</comment></comment><comment><USER11> First of all, the question you need to ask yourself is what type of leverage do you have in this situation and what type of relationship do you want to keep with this company you are at.

In terms of leverage, if you stop and quit right now, what will happen to the project? Are you on the critical path? This does give you some leverage.

Also, what level of documentation do you have with regards to getting paid? Based on your statements, you have met with management and have had verbal discussions. I would suggest doing these things via email. Do not just leave it to verbal communication for obvious reasons.

At this point, you need to build your case for yourself to prove that the company owes you money and you have not been paid. This is not so much that you would sue but that if you do go down that path you have evidence.

The next step would be to write an email to management stating:

1) The work you have done and how much you are owed. 2) A deadline expecting to get paid (lets say a week). 3) What will happen if the deadline is past and payment hasn't been made.

Be prepared for a couple of scenarios in this case. The company ignores your email and the deadline is past. You need to walk away. How nasty do you want to be is also a considering. You can backup your work and remove it from source control and hold the company for ransom until you get paid (I know someone that did this and they got paid pretty quick). If you don't want to go down that road, quit and look at suing the company with the expectation that you may not recover your wages.

The other scenario is that the company does pay you. In this case, take the money and find another job pronto. There is no reason to work for a company like this.

The other thing I want to add to this is that it is not your company and not your problem that the client is not paying the company. Don't make it your problem.

The most important thing is that you need to stop working without getting paid. If you feel the company is going to screw you over, just quit and spend that time looking for another job. Your payback with the company can be that you spread the word in the area that they a deadbeat company and everyone should stay away.

</comment><comment><USER12> You should lawyer up as others advise, but I must be missing something because the problems you listed are definitely not the client's problems. For example, it isn't their problem you didn't filter candidates well enough, etc.

I would lawyer up, but also work on killing them with kindness. Make it right. Gary-Vaynerchuk the heck out of them. ([http://www.garyvaynerchuk.com/](http://www.garyvaynerchuk.com/))

</comment><comment><USER13> similar topic: [https://news.ycombinator.com/item?id=6353498](https://news.ycombinator.com/item?id=6353498)

</comment><comment><USER14> go to the next meeting with a baseball bat</comment>


Astronomy Picture of the Day – SN Candidate Betelgeuse to Scale (2020) - chupa-chups https://apod.nasa.gov/apod/ap200101.html <comment><AUTHOR> Editorialized title: real title is "Betelgeuse Imagined".

But the real lede here is buried: "Betelgeuse, one of the brightest and most recognized stars in the night sky, is only half as bright as it used to be only five months ago." This is actually quite shocking to me, I'm very surprised I hadn't heard about it until now.

<comment> chupa-chups Actually, the real title is

> A plume on Betelgeuse

Nasa itself links to [https://www.eso.org/public/images/eso0927d/](https://www.eso.org/public/images/eso0927d/).

<comment><AUTHOR> See the guidelines: [https://news.ycombinator.com/newsguidelines.html](https://news.ycombinator.com/newsguidelines.html)

"please use the original title" -> in this case that means "APOD: 2020 January 1 - Betelgeuse Imagined" which is literally the page title. I think in this case "Astronomy Picture of the Day: Betelgeuse Imagined" would be acceptable.

</comment></comment></comment><comment><USER2> I'm hoping it went supernova sometime around 1320 CE to 1370 CE.

(It's about 700 lightyears away)</comment>