Time to tackle the real evil: too much debt - <AUTHOR> http://www.ft.com/cms/s/0/4e02aeba-6fd8-11de-b835-00144feabdc0.html?nclick_check=1

<comment><USER2> So I'm somewhat confused with the article's suggestion that:

"The only solution is to transform debt into equity across all sectors, in an organised and systematic way. Instead of sending hate mail to near-insolvent homeowners, banks should reach out to borrowers and offer lower interest payments in exchange for equity."

As I understand it, most that will walk away from their homes are upside down, meaning they owe more on the property than it is worth. That is also the reason that they're unable to refinance.

So the authors suggest that the bank lower the interest on the loan in exchange for equity which the homeowner has none of?

<comment><USER3> Imagine you bought a million dollar house, with nothing down (for simplicity's sake).

It's now worth $900K and you're considering walking away.

What if the bank took 15% ownership of your house, in exchange for reducing the principal balance to $800K? You'd pay less on your mortgage (balance of $800K instead of $1000K), and while you'd be slightly upside-down (you'd owe $800K on $765K worth of house), you're still much closer than when you were $100K in the hole. (Plus, you don't have the PITA, stigma and credit rating destruction of getting foreclosed on.)

Fast forward a few years and you can sell the house for $950K and decide to do so. You get $807K, the bank gets $142K (compensating them partially for forgiving $200K of your original balance in exchange for the 15%). In almost all cases, that delayed $58K loss is better than an immediate $250K loss from foreclosing and selling for $750K or so.

That's but one way to accomplish the debt-to-equity exchange.

<comment><USER4> > Fast forward a few years and you can sell the house for $950K and decide to > do so. You get $807K

Hmm - no. You get a little bit over $7k. You still have to pay off the mortgage, and very little of that happens in the first few years.

Note that you're assuming another housing boom because you need it to make the "equity forgiveness" look less insane.

The big problem is the assumption that these folks can pay an $800K loan yet a $1M loan is completely out of their range. For a small minority, maybe but most of the people in this situation can't afford the $800K loan either.

Yes, some folks are bailing on loans that they can afford, but the vast majority of foreclosures involve people who can't afford their loans. They're not 20% away from making it, they're 50-70% away. (Banks don't foreclose on folks who are current.)

<comment><USER3> 6% housing appreciation in a few years is FAR from another housing boom. In fact, it's under most estimates of inflation. And with the way we're printing money over here, I think inflation is far more likely to increase than decrease.

I don't think that this is necessarily a "great idea" but I was just trying to explain how the bank can get "equity" (a share of ownership) from a homeowner who is underwater in their current mortgage.

<comment><USER4> > 6% housing appreciation in a few years is FAR from another housing boom. In > fact, it's under most estimates of inflation.

If it's under inflation, this equity deal is a bad one for the bank.

> I was just trying to explain how the bank can get "equity" (a share of > ownership) from a homeowner who is underwater in their current mortgage.

No - you were claiming that this sort of deal made sense for a bank. Getting equity isn't actually a problem. And, there are many deals that a bank can do, so inventing one isn't actually all that interesting.

The relevant deals are profitable.

And, you still haven't addressed the real problem, that the vast majority of these folks can't afford an $800K loan either.

<comment><USER3> > If it's under inflation, this equity deal is a bad one for the bank.

The bank is already in a tough spot. They're $100K+ in the hole in a foreclosure situation, so going a lesser amount in the hole is an improvement in their situation. You cannot compare to the "don't make the original mortgage loan case" and say that because they're "giving away" $65K in value that it's the worst of their available options.

> No - you were claiming that this sort of deal made sense for a bank.

Please show me the text that makes that claim. (I happen to think that it does, but I don't believe I actually made that claim.)

> And, there are many deals that a bank can do, so inventing one isn't > actually all that interesting.

> The relevant deals are profitable.

There are many. OK, please propose maybe two alternate deal structures under the above facts/circumstances that are materially different and better for the bank. (Saying they'll take 18% equity for otherwise the same loan mod is not materially different IMO, even though it's obviously better for the bank.) Ideally your proposed deals would have a chance in hell of the bank offering and the borrower accepting.

> And, you still haven't addressed the real problem, that the vast majority of > these folks can't afford an $800K loan either.

The fact that people who once got a million dollar mortgage and now can't afford an $800K mortgage is not my problem to address. I didn't loan them the money, and unless they're in my neighborhood, it's such a vanishingly small part of my damn business or concern that I'll choose to spend time on things that do matter to me or that I can influence.

If the bank lent to someone who couldn't possibly repay on the day they took the loan, that person needs to get foreclosed on, the bank needs to take the losses and look at its underwriting standards, and the borrower should have their credit rating dinged for their own irresponsibility.

If a job loss or income change prompts the foreclosure, then I feel more sympathetic to both the lender and borrower, but some portion of loans are going to go bad, and in the "less than 20% down" loans, the bank was knowingly taking larger risks, so I don't feel too bad about their larger losses. The borrower in this case has still made and failed to keep a commitment, which speaks to creditworthiness, so again the credit downgrade is fair/appropriate, even if it seems like "piling on".

For people who could afford an $800K mortgage (and many of them could afford to keep current on their $1mm mortgage), but there are some who would choose not to because their mortgage is non-recourse and they're far enough underwater on the house that it's literally in their best financial interest to walk away. Those people are the ones for whom a loan mod could make more sense to both sides than a bank foreclosure.

<comment><USER4> > Please show me the text that makes that claim.

You wrote that it was better than foreclosure.

> The fact that people who once got a million dollar mortgage and now can't > afford an $800K mortgage is not my problem to address.

Technically, none of this is your problem. However, you claimed to be offering useful suggestions for banks' problems. Trading a loan that doesn't work for a loan that can't work doesn't help the bank.</comment></comment></comment></comment></comment></comment></comment>


Forecasting at Uber with RNNs - <AUTHOR> https://eng.uber.com/neural-networks/ <comment><USER2> I wish the diagrams were bigger, they are hard to read and a bit blurry.

One of the interesting points, that is often overlooked in ML is model deployment. They mention tensorflow, which has a model export feature that you can use as long as your client can run the tensorflow runtime. But they don't seem to be using that b/c they said they just exported the weights and are using it go which would seem to imply you did some type of agnostic export of raw weight values. The nice part of the TF export feature is that it can be used to recreate your architecture on the client. Bu they did mention Keras too which allows you to export your architecture in a more agnostic way as it can work on many platform such as Apples new CoreML which can run Keras models.

<comment><USER3> Warning: I'm a vendor. Take everything I say with a grain of salt. I will try to sell you something.

1 biased perspective I have here: Infra is often a different team from data science. They don't always do the deploying. Beyond "some sort of serving thing" the data scientists might not necessarily know about what's being deployed. This is not true at every organization and there are exceptions. This is typically true of most companies we sell to though. There are usually ML platform teams that do the "real" deployment (especially at sizable scale)

Another characteristic of production is it's "boring". "Production" is a mix of databases to track model accuracy over time, possibly microservices depending on how deployment is "done". Characteristic ways of giving feedback when a model is wrong, experiment tracking and model maintenance among other things.

A lot of these things are typically very specific to the company's infrastructure.

The "fun" and "sharable" part that people (especially ML people) is usually related to "what neural net did they use?"

The other thing to think about here: "production" isn't just "TF serving/CoreML and you're done" there's typically security concerns, different data sources,.. that are often involved as well that might be specific to a company's infrastructure. There also might be different deployment mechanisms for each potential model deployment: eg: mobile vs cloud.

Grain of salt sales pitch here: We usually see the "deployment" side of things where it's a completely different set of best practices that happen to overlap with data scientists experiments. This includes latency timing, persisting data pipelines as json, gpu resource management, kerberos auth for accessing data, managing databases and an associated schema for auditing a model in production (including data governance), connecting to an actual app/dashboard like the ELK stack,..

TLDR: The deployment model would be its own blog post.

<comment><USER4> The Google paper _Machine Learning: The High Interest Credit Card of Technical Debt_ [1] offers a semi-rigorous introduction to the topic of _real-world_ ML model engineering/deployment considerations and best practice. (If anyone else knows of similar work I'd be grateful to hear about it.)

[1] [https://research.google.com/pubs/pub43146.html](https://research.google.com/pubs/pub43146.html)

<comment><USER3> This is actually a great reference! Thanks for the link.

</comment></comment></comment></comment><comment><USER5> I wonder how much they could enlist others to solve this by creating something like an 'Uber Auction House' to basically buy and sell the right to reap Uber's cut for a ride. They could clean up on exchange fees while everyone solves this problem for them.

<comment><USER6> This is interesting, could it potentially reduce surge pricing?

One thing I thought, it would be really convinient if Uber could amortize their surge pricing over the month/year. In order not to hit customers with unexpected rates and essentially offer a flat predictable fee over the whole period. Problem with that is you can't really plan the future demand to calculate how much you need to save/dip. Could an auction house help to hedge the bets?

<comment><USER7> It's unlikely to affect surge pricing. You can think of surge as a tool to move drivers to where the riders are. Even if you knew there would be heavy demand, you need to incentivize drivers to actually go there. Why would I, as a driver, go 20 minutes out of my way to be in Katy Perry concert traffic when I can keep picking up passengers at a reasonable clip on the other side of town?

Surge is only really solved once autonomous vehicles can be preemptively positioned near demand.

<comment><USER8> The author might be suggesting that you could have surge priced compensation for the drivers to incentivize them to move to the demand but also amortize that cost for the consumer.

<comment><USER7> That's an interesting problem because (if I'm understanding you correctly) forecasting would need to be done at an individual level. A consumer getting rides primarily during off-hours should (imo rightly) pay less than a rider booking rides primarily from busy locations. Amortizing that cost fairly while also making a safe profit is a tricky balance to strike, I'd think.

</comment></comment></comment></comment></comment><comment><USER9> I don't understand if they use windowing as a fixed computational step that is active both in training and scoring time, or, if they use sliding windows only to chop up the training data.

Also, I wonder if they checked how a feed-forward NN that operates on the contents of a sliding window (e.g. as in the first approach above) compares with their RNN results. I am curious about this, as it would give us a hint whether the RNN's internal state encodes something that is not a simple transformation of the window contents. If this turns out to be the case, I'd then be interested in figuring out what the internal state "means"; i.e. whether there is anything there that we humans can recognize.

[edited to increase clarity]

<comment><USER10> I wasn't very sure what the sliding window part was about either. I think they were just saying that they trained on a sliding window using the "output window" as part of their loss function.

A feed-forward NN wouldn't do much because it doesn't hold a state variable which you need to be able to understand context in time series data. There are probably some pieces of the state that you'd be able to interpret but the majority of it would mean nothing to us.

</comment></comment><comment><USER11> Whenever I see a post or announcement by a major company that they're using "machine learning", I'm reminded of what CGP Grey said: it seems like nowadays machine learning is something you add in to your product just so you can seem hip by saying that it has machine learning, and not for a legitimate technical reason.

There are undoubtedly things that machine learning is right for, however to me it seems like it's become a buzzword more than anything else.

<comment><USER12> Resume-driven development contributes some momentum here, too.

<comment><USER13> Resume-driven development - never heard that term before. Now I have an excellent descriptor for what I see all too often.

Maybe O'Reilly should release a book on RDD?

<comment><USER12> Resume-driven development is an organizational pattern, and like all patterns, it exists for better reasons than you'd first think.

The lifetime compensation of developers isn't just tied to how much salary they have at the moment. Getting into a dead-end and not developing your skills will definitely set you back over the long term. Like, there's a reason people will pay you a premium for working with COBOL. So there's a very rational pressure to get better total compensation out of a role by choosing resume-developing tools and technologies.

On the other end, organizations mostly just care about getting the task done. They've got the choice of doing it in a boring way and paying lots of money for developers who don't want to grow their resume, or indulging in the developers fancies and getting it done more cheaply.

tl;dr - resume driven development is a way for companies to pay for projects with "you'll get experience".

</comment></comment></comment></comment><comment><USER14> Interesting stuff, but all they've managed to do so far is find models that fit historical data better. Would be interested to read a follow up a year later to see how their models actually performed.

</comment><comment><USER15> I wonder how are they quantifying uncertainty around their predictions. Having a point-estimate without some notion of confidence interval seems much less useful. Is there a natural way to do this through LSTMs?

Also, some actual benchmarking would be great. Say, against Facebook's Prophet (which also deals with covariates and holiday effects).</comment>


Which are the best training institute for learning Big Data and Hadoop? - <AUTHOR> <comment><USER2> I think that would be the institute paying you to post this "question".

</comment><comment><USER3> Which part of the world?</comment>


The Smash Community Is Chaos Right Now - <AUTHOR> http://kotaku.com/the-smash-community-is-chaos-right-now-1746040568 <comment><AUTHOR> Note: article is more about development than gaming.</comment>


Man Poses as Woman on Online Dating Site; Barely Lasts Two Hours - <AUTHOR> http://jezebel.com/man-poses-as-woman-on-online-dating-site-barely-lasts-1500707724 <comment><USER2> I was part of the team that rewrote Finland's most popular dating site about two years ago and this was one of the problems we tried to fix.

We got pretty good results by letting users block messages from users who do not meet certain criteria. We gave users three different filters: 1\. Allow messages from users who live in those areas where you are looking company 2\. The same as 1., but also age and gender must match your "dream partner" settings 3\. And finally allow messages from users who meet X percent of your dream partner settings

And of course you could block users after they have messaged you, but that is already too late if you get 100 harassment messages per day.

<comment><USER3> why not block a user from sending messages to anyone after, say, 3 complaints? yes, maybe it's a little harsh and not democratic, but it will make men much more considerate

<comment><USER2> Such a system was built for images; if user images got too many complaints, then those images were hidden and sent to moderation queue where moderators could decide wheter those complaints were valid or not.

It would have been fairly simple to do same for messages, but problem is that Finnish law does not permit moderators to read private messages. So it would have been unfair for some users.

</comment></comment></comment><comment><USER4> The very first thing I did as a teen when I tried the internet in '95 or '96 was to pretend to be a woman in a chatroom (I realise how weird that must sound). That brief experience was a valuable lesson and taught me how not to talk to women.

Still, I'm very much a typical horny male – it often takes no more than about 2 seconds to go from noticing a woman anywhere to mental procreative scenery. For all the feminist talk about the female libido, I doubt women generally experience as strong (or perhaps "swift" is a better word here) urges as many men seem to do. This discongruence is perhaps one of nature's cruelest jokes... The question is how to avoid demonising (all) men without victimising women.

However, before this whole discussion turns into another bitter (anti-)feminist warzone, this particular case has probably at least as much to do with the medium as it has with issues between men and women. As it stands, the (unwalled) internet is generally somewhat of a lost cause since the distance, anonimity and lack of social consequences make it very hard to compel people to do the right thing – at least according to current societal norms.

If you prefer the wild west over the walled gardens of strictly regulated sites (block, ban, ignore, real ID), I'm afraid you better bring a gun to handle the coyotes... Not many seem to be in favour of policies such as Real Name or let alone forms of official identification.

</comment><comment><USER5> There's a subset of men who enjoy sending sexually explicit messages to strangers. This is not surprising. Depressing, but not surprising.

What _is_ slightly surprising is that online dating websites, for all their sophisticated matching algorithms, haven't bothered applying some of the most basic anti-spam heuristics to the problem. The first dating site that does it, you'd imagine, would get a very good reputation very quickly.

Dating websites actually want people to stay single and remain on the site paying their subscription or viewing adverts, but an add-on that filters out the awful rubbish is _surely_ something women would pay for.

<comment><USER6> The problem is making a filter than works for everyone. A 50 something conservative christian woman probably has different ideas about what constitutes as acceptable message to a 20 something liberal woman looking for a more casual fling.

<comment><USER7> Well, there's always filter settings.. let everyone rate a few messages into acceptable or not and go from there. (Maybe go from real inbox messages in case one wants to minimize additional exposure.)

<comment><USER6> You need a pretty large corpus to train a filter effectively, so probably not practical to train on a per-user basis.

<comment> loup-vaillant The filter doesn't need to be accurate right away to be effective. Also, you could design a training set that gives more information than something made up of random messages.

Anyway, if the filter weeds out most spam without getting too much false positives, that's a win. Then, as you mark more messages as "inappropriate", you get less and less such messages.

<comment><USER6> As anybody who's ever tried to train an email spam filter from scratch can tell you , the number of messages you need to classify in order to be useful is _huge_.

With email spam, everyone can more or less agree on what it looks like so you can reuse other people's training data but by deciding on what is a "bad" online dating message is much more complicated.

<comment> loup-vaillant Okay.

</comment></comment></comment></comment></comment></comment></comment><comment><USER8> Someone should try a similar experiment: "Unemployed Homebuilder poses as Software Engineer on Jobs Site, OMFG too many recruiters".

Or "Poor Indian wears Fair&Handsome and poses as Rich Western Tourist, barely lasts two hours before getting annoyed by beggars".

Related thoughts from the often interesting Scott Alexander: [http://squid314.livejournal.com/327849.html](http://squid314.livejournal.com/327849.html) [http://squid314.livejournal.com/327957.html](http://squid314.livejournal.com/327957.html)

(Yes, I'm one of those big jerks who has as much sympathy for the sexually deprived as the monetarily deprived, which is not a huge amount. Feel free to make status-lowering comments in response.)

<comment><USER9> I don't get it. Let's take your software engineer example. Software engineers often _do_ complain about spam from recruiters, in cases where the recruiters:

a) Plainly have not taken your skills or location or pay grade into consideration

b) Refuse to take "no" for an answer

c) Turn hostile or somehow imply that you owe them your time or should be grateful for their attention

And we generally assume that the engineers are correct in their complaints. I can't say I've ever seen recruiters being abusive, but I have heard voicemail messages that start with "I know you said not to call you again, but...", and there's really no question about who is in the wrong there.

Whether it's actually _important_ is a different question. I get a fair few pointless enquiries via LinkedIn, and I ignore them. I can't say that it troubles me much. But if the people sending those messages were insulting me, making sexually explicit statements or generally being persistent beyond all efforts to make them stop, I would probably feel much more strongly about it.

What puzzles me is that the behaviour of the recruiters does make at least some sense in a way that the dating site behaviour doesn't. Sure, getting a job spec for a senior Java developer and then mailing everyone with "senior [something] developer" in their past job titles on LinkedIn is going to fail quite often, as you'll hit Senior Python Developers and Senior C++ Developers and so on, but sooner or later the recruiter has to get lucky. But who the hell responds positively to the kinds of comments that the OP received? Are the people sending the messages doing so because they believe that sooner or later it's going to _work_? Even worse, _does it actually work sometimes?_ I can't easily believe that it does, but it's hard to understand the behaviour otherwise.

<comment><USER8> I complain about recruiters. I also complain about fat girls/gays (note: not comparing gays to fatties) hitting on me, touching me in ways I dislike, not taking no for an answer and cockblocking me when I talk to people I find more attractive.

But I don't adopt the same victimhood that the Jezabel blog post is trying to imply, i.e. I won't favorably quote people making statements like this: "I came away thinking that software engineers have it so much harder than buggy- whip makers do when it comes to that kind of stuff..."

<comment><USER9> Fair enough, in principle. However, I think the OP is saying that there is a difference in the _nature_ and the _quantity_ of the comments such that it crosses the line from mild annoyance to something worse; it's so off-putting that it prevents the women in question from participating in online dating sites. And, since the women are an important component of dating sites, it's probably important to take that into consideration when designing such sites.

If you're happy being a bit more stoical about the problems you've encountered, that's fine. Perhaps some of the women receiving unwanted messages on dating sites could be a bit more stoical too. But it does seem to me that they're being given a bit more to be stoical _about_ , and this is unfair, to some extent (it is perhaps hardly the greatest injustice in the world, but it does seem genuinely unfair).

</comment></comment></comment></comment><comment><USER10> Call me jaded but I suspect OKCThrowaway22221 was in fact a woman, posing as a man, posing as a woman.

<comment><USER11> If you wanted to be more or less confident of this, you could try running some of what (s)he wrote through that application that guesses your gender from how you write stuff.

EDIT: I put the paragraphs quoted into [http://www.hackerfactor.com/GenderGuesser.php#Analyze](http://www.hackerfactor.com/GenderGuesser.php#Analyze)

        Genre: Informal
          Female = 1683
          Male   = 1466
          Difference = -217; 46.55%
          Verdict: Weak FEMALE

        Weak emphasis could indicate European.

Site carries this disclaimer: The system generates a simple estimate (profiling). While Gender Guesser may be 60% - 70% accurate, it is not 100% accurate. This is better than random guessing (50%), but should not be interpreted as "fact".

<comment><USER12> Wow. I'm stunned by how well that works. I've copy pasted 5 male and 5 female paragraphs, and it got every single one correct.

<comment><USER11> The last time I used it in earnest I did not have anywhere near the results you are describing. So my own personal perception of it is that it's probably as accurate as they say it is. More than 50% but less than 60-70%.

<comment><USER13> Are you American? They say that their training set has an American bias (reason for the "may indicate European ..." disclaimer).

</comment></comment></comment></comment></comment><comment><USER14> I was part of developing a Swedish dating app, and during this time I discussed online dating with many friends of both genders. While many women would find the attention a bit tiring, and sometimes had to report someone for the tone in their messages, no one ever told me anything like this. I would guess that it heavily depends on which kind of dating site you use. So to counter this story, I share that I've heard 10-15 stories that was much more positive.

</comment><comment><USER15> How did Jezebel get unbanned?

Aren't they part of the Gawker/Valleywag hive of scum and villainy?

<comment><USER16> I don't know but I can tell you I stopped reading at "teh poor menz." Do people still do that "teh" thing?

</comment></comment><comment><USER17> ☑ Don't show commonly blocked users and prevent them from contacting you.

^ This would be an attractive feature.

<comment><USER18> Even better , ghost them and make them pay for each message they send... (kidding,as it would be fraudulent).

</comment></comment><comment><USER19> I love the topic, bur god, Jezebel's writing is painfully atrocious.

<comment><USER20> Unbelievable isn't it? It used to be a good source of interesting articles about gender/sexual politics but at some point they must have worked out that the excessively conversational "new york" style articles get more traffic, and now you can almost hear the exposed brickwork in every sentence.

</comment></comment><comment><USER21> I have some friends who got married via an online dating site, but it was a paid one. I get that this problem plagues free sites like OKC or POF - is it also a problem on paid sites?

<comment><USER22> > I have some friends who got married via an online dating site, but it was a > paid one.

Note that it's not a post against online dating sites, commenters on both 2x and jezebel note that they've met significant others on dating sites.

> I get that this problem plagues free sites like OKC or POF - is it also a > problem on paid sites?

Creepy messagers have as much money to spend as others (if not more) I'd guess, unless the site finds out they make people leave (and stop paying) I'd expect little difference. But you'd have to ask women using or staff of these sites to know/get a comparison really.

</comment></comment><comment><USER23> My wife and I met on an online dating site. Mainly because we were chatting on the open forum about how weird everyone was, and how fed up with it we were.

</comment><comment><USER24> Matching couples looks like an easy problem to solve, here, you have my little algo to find the optimum match for n couples: [https://github.com/kfk/scripts/blob/master/match- date.py](https://github.com/kfk/scripts/blob/master/match-date.py)

Ok, joking a bit, but there is a point to this. If you have a finite set of people (and I bet in a short time frame you can make that assumption), then in the WORST case you have to do n*(n+1)/2 matches. Let's say it takes 1 hr to run a date, then we can match 10 people in 55 hrs in the worst case.

Now, try matching 10 people using these websites. Enter the texting game and no proper filtering (by looks, age, location, etc.). It will take X time my worst case scenario to maybe match half of the population.

Point is: somebody should make a system that clusters people and randomly assigns them to dates. Without any prior texting that is. Which, by the way, would simulate the way people met until 20 years ago more or less.

</comment><comment><USER25> Reading this is interesting both from the content it describes (a guy passing for a girl on a dating site and being horrified by the messages he gets) and from the tone, that all this is self-evident and why don't guys just know this? Well, they don't. But when they learn it, they are as horrified as it deserves.

</comment><comment><USER26> sounds like a good opportunity for dating sites to do pre-moderation of inbound messages. I am sure some of it can be caught with a relatively simple classifier (machine learning) and the rest can be handled with mechanical turk.

Another approach would be to have a karma system that over time will filter out the bad actors.

<comment><USER27> You want to send private messages to mechanical turk?

Not a good idea. Karma would work (but could be abused).

<comment><USER26> yea, good point about Mechanical turk. Moderators would get expensive quite quickly but if it is such a royal pain I suppose it can be a paid-for feature which is sustainable.

</comment></comment></comment><comment><USER28> Well I don't know about you guys but that isn't surprising to me at all.

It's amazing how big the mountain of stupid is out there and it's not easy when their all trying to talk to you.

That's why us guys have it hard in general. When she has to sift through so many emails to find just a half decent guy.

Lesson 1: You need to stand out if your just like the other 20 guys she's talked to these past hours you're doomed to fail. Be yourself but be your best self and don't be afraid to be unique or weird as long as its not in a creepy way.

Lesson 2: Don't be a wuss or needy let her reply in her own damn time and if you're writing a follow up be funny and bust her balls a bit for not replying in a timely manner. If she still doesn't reply ...oh well there's plenty of other fish in the sea.

<comment><USER22> I'm impressed by your skills at completely missing the point, victimisation reversal and turning a discovery/report of many women's day-to-day creepy and harassy experiences (on dating websites specifically, but the 2X comments make it clear this extends to online and off-) into "yeah mates it's hard for a guy to get noticed, here's one weird trick to be picked"

<comment><USER28> If you discovered a cure for guys being horny and stupid go ahead I'm listening.

This is not something new women have been dealing with this since the birth of the internet and there's nothing you can do about that.

All you can change is yourself and learn from other guys stupidity.

</comment></comment></comment><comment><USER29> In online dating, and human interaction in general, you need 2 basic things.

#1: People to talk to.

#2: The person need to a person you want to talk to, ie not a "predator", scam artist, bot, creepy, and so on.

The question the person had was if women had a easier time on dating services than men. By experiment, he concluded that the mass of men wanting to talk to a female profile and the over predatory attitude quickly overwhelmed the person who made the fake profile. By this, he concluded that women has a worse time than men.

Just me who do not think he actually answered his question?

</comment><comment><USER30> Why not limit the amount of messages people can send per day? 3 for instance? Wouldn't that make you think before sending a message? Maybe users could choose to only receive messages from people who send less than N messages per day.

It would be funny if messages from users who are reported as annoying get redirected to a horny bot which keeps them "entertained" :)

</comment><comment><USER31> Why didn't he just click "block" and wait for normal people to contact him? There are probably some idiots who jump onto every new profile that pops up. Once blocked, it will become much easier.

<comment><USER32> Do you remember when spam filters didn't work?

</comment></comment><comment><USER33> What about a dating website that allows each guy to send only one message per day?

</comment><comment><USER13> Link to Reddit thread:

[http://www.reddit.com/r/TwoXChromosomes/comments/1uqym6/as_a...](http://www.reddit.com/r/TwoXChromosomes/comments/1uqym6/as_a_guy_i_wanted_to_know_what_it_was_like_to_be/)</comment>


"Extremely angry with the state of academic CS research right now" - <AUTHOR> http://neverworkintheory.org/2016/04/26/perception-productivity.html <comment><USER3> For an engineering community manager, that was an awfully rant-y way to make the point: "It's frustrating that researchers don't publish their code."

Not to say I disagree with the frustration... but it's also not something new. It's been this way for decades. I'd much rather hear about who _is doing_ work in this space and what they're working on. Here are the ones I know of:

1\. _The Center for Open Science_ ([https://cos.io](https://cos.io)) is one such org trying to fix this with the Open Science Framework [1].

2\. _GitHub_ also recognizes the need for citable code and gives special discounts for research groups, in fact, Mozilla is one they work with [2].

Two smaller related startups are:

3\. _Datazar_ ([https://datazar.com](https://datazar.com)) - A way to freely distribute scientific data.

4\. _Liquid_ ([https://getliquid.io](https://getliquid.io)) - A scientific data management platform. Somewhat like "Excel for scientific data as a Service".

Also, a related HN thread from some years ago: "We need a GitHub of Science" [https://news.ycombinator.com/item?id=2425823](https://news.ycombinator.com/item?id=2425823)

\---

1: [https://osf.io](https://osf.io)

2: [https://github.com/blog/1840-improving-github-for- science](https://github.com/blog/1840-improving-github-for-science)

<comment><USER4> I like his rant. But at the same time, as an ex-CS grad student now doing biology, I wish my field was 1/10th as rigorous, tidy and beautiful as computer science.

Code is a joke, most data is processed using "pipelines", which in reality means some irreproducible mess. People don't generally do research trying to understand how cells or tissues work, they generally write papers about "stories" they found. Only a small minority are trying to do some serious modeling using serious math.

<comment><USER5> > Code is a joke, most data is processed using "pipelines", which in reality > means some irreproducible mess.

You're not wrong, and it's not limited to bioinformatics; Reinhart-Rogoff's findings were reversed when an additional 5 rows were included in a spreadsheet they used to calculate their correlation between GDP growth and debt ratios. And of course, they insist that despite the actual outcome being _twice as strong and in the opposite direction_ , they still support their original position.

I wonder if one can get a CS PhD by producing enough retractions. Of course, it won't win you many friends in the academy, and would probably lead to less source code made available. But given the Perl code I've seen published who's termination condition is a divide-by-zero exception, one can argue that peer review in the information age has to include code review.

<comment><USER6> Didn't know about the Reinhart-Rogoff controversy [0], interesting! They state that they have been _careful not to claim that high debt causes slow growth, but rather that it has an "association" with slow growth_.

[0] [http://www.huffingtonpost.com/2013/04/16/reinhart-rogoff- aus...](http://www.huffingtonpost.com/2013/04/16/reinhart-rogoff-austerity- research-errors_n_3094015.html)

<comment><USER7> I read abt this quite a bit, the refutation is attacking a small portion of the data, because that small portion is trendy and hot in politics.

Judging academically, the original paper and the refuting paper is a healthy debate, but the dynamic of the society and politics ab-use them to attack a whole school of thought at large (the austrian school: less bailout, less intervention by government, less control over everything) in favor of keynesian school (more bailout, more government spending, more public debt, especially in recession and crisis).

Anyway, it remains a controversy, because theoretically one can do what one wants, but once it involves policy and real life matters, it is hard to argue for what method is right and what is wrong, in the presence of so many (ready to b angry) interest groups.

<comment><USER8> Excuse me? That single attacked paper was the intellectual blanket for an unprecedented victory march of the Austrian school after the financial crisis and the recession.

I agree that from a purely academic point of view this is nothing big to worry about, but this paper played a completely outsized role. And the authors stood by and let things run their course, without any attempt to reign in or moderate the debate.

[https://en.wikipedia.org/wiki/Growth_in_a_Time_of_Debt](https://en.wikipedia.org/wiki/Growth_in_a_Time_of_Debt)

<comment><USER7> Fair enough, but i add that academic life is sad, one has to pursue one's endeavor at one's own cost. However, politicians and the public want too much from us researchers. So sometimes, we do believe that our sweating formulas have life impact, or to fancy, save the world.

</comment></comment></comment></comment></comment></comment></comment><comment><USER9> How is this different from any other academic research? What he is asking about is neither openness nor reproducibility (which are, indeed, very important). He is asking that researchers produce code that he can put into production. Not only they have negative incentives to do so (for one, providing such code will surely result in a stream of all kinds of support requests), it would actually work against the reproducibility objective.

The purpose of the code written is usually very simple: to produce results of the paper, not to provide a tool other people can use out of the box. Even when such a tool is nominally provided (for example, when a statistics paper is accompanied by an R package), there are good reasons to be very careful with it: for example, the paper may include assumptions on valid range of inputs, and using the package without actually reading the paper first would lead to absurd results -- which is something that has happened. The way to use academic research results is to (1) read and understand the paper, (2) reproduce the code -- ideally, from scratch, so that his results are (hopefully) unaffected by authors' bugs, (3) verify on a test problem, and (4) apply to his data. Using an out of the box routine skips steps 1-3, which are the whole point of reproducibility.

<comment><USER10> > reproduce the code -- ideally, from scratch, so that his results are > (hopefully) unaffected by authors' bugs

This rests on a common false assumption that programmers make: they think it's easier to write bug-free code when starting from scratch. The reality is that it's almost always easier to start with something that's nearly working and find and fix the bugs.

What really happens when you do a clean room reproduction is that you end up with two buggy programs that have non-overlapping sets of bugs, and you spend most of the effort trying to figure out why they don't match up. It's a dumb way to write software when it can be otherwise avoided.

<comment><USER11> I wonder though, maybe non-overlapping sets of bugs are actually better for science? That is, it could avoid systematic errors. Of course, one bug free implementation is clearly better!

</comment></comment></comment><comment><USER11> Two points to consider here. First, it's not fair to criticize the whole field of academic CS research. Not everyone's work can be accurately represented by a github repo. Second, even when it can, expecting the "Run author's [whatever] against an up-to-date dataset." step to work is asking a quite a lot. Typically there are a infrastructural assumptions baked in (file paths, legacy code dependencies, etc) and manual steps to get to plotted results. In an ideal world with enough resources, every lab would have technical staff to help with this process, but most researchers unfortunately don't have bandwidth to spend on this problem.

<comment><USER12> If your paper has results, it has code of some sort which can be put into a github repo.

That's the bare minimum. If you don't know how to make code agnostic to file paths or dependencies, that's too bad, but fortunately a field practitioner picking your code up _will_ know how to work around those issues. At least they're not starting from scratch on trying to rewrite your code.

<comment><USER13> As a theorist, not it goddamn doesn't.

<comment><USER14> How do you validate your theories?

<comment><USER13> I prove them.

</comment></comment></comment></comment></comment><comment><USER15> On this topic I'd like to mention the machine learning papers site GitXiv, which is wonderful. It publishes papers alongside the Github repo containing the code.

[http://gitxiv.com/](http://gitxiv.com/)

<comment><USER16> Thanks! That's awesome. I have long thought that a platform for scientific publication where code and data are included would be valuable, and I'm glad to learn it exists. Going further, imagine the code is like a container that anyone can run to reproduce the findings, including statistical analysis and summary on raw data sets -- the key findings should "build". Perhaps the platform also provides continuous build during "development" (research) so that researchers can work privately and then publish their "repo" publicly along with their paper. An easy way to clone and reproduce the build after publication: "fork my research"

As an extreme version of the idea, imagine if the actual paper itself (TeX) and all the data within it are also built as part of the repository; any graphs in the paper are rendered from data in the repo, any numbers are data accesses, etc. This probably wouldn't be helpful to researchers, but it would promote scientific reproducibility and aid everyone building on a researcher's work. Tremendous work goes into authoring the papers themselves, sometimes with methods or tricks that are private; laying it all out publicly would greatly help students of science.

Going even further: to avoid cherry picking of positive results, review boards expect experimental criteria to be published (at least privately to them) in advance, for research that involves capital-E experiments. Perhaps this includes analysis code at least in prototype form; like test driven development, the acceptance criteria are written first. When the paper is ready for review, the reviewers can compare the initial prototype analysis logic to the final form. Perhaps the board also expects all data and trials collected during experiments to be made available in the repository, whether positive or not. All collected data should be in the platform, in the most raw form it was originally recorded, as well as all steps of summary and analysis.

I wonder if a process and platform like this could contribute to the integrity and quality and reproducibility of scientific research. People funding research ought to ask for it, especially public funded research, and the whole repo is made open eventually if not initially.

Perhaps as part of the platform's value prop to researchers (on whom it is imposing probably more burdens than benefit, for sake of _public_ benefit), the hosting is free and funded by a foundation, or steeply discounted. (OK, it won't pay for LHC scale data sets, but otherwise ...) So using it to host your data, code, and paper is free, at least up to a point. I would be interested to contribute time and resources toward building or supporting a platform like this.

<comment><USER17> I don't think research should be as structured as software development. In CS, many of the most interesting papers come about when the authors discover something unexpected or non-intuitive and choose to explore down that thread. That's why it's research -- sometimes you can't know what you will find until you're there.

<comment><USER18> _I don 't think research should be as structured as software development_

In many ways, it already is: good research requires meticulous log keeping in order to reproduce results, and equal effort must be spent on maintaining references to other literature, or you risk missing a citation in a published paper.

</comment></comment></comment></comment><comment> rudi-c I took a graduate course on software engineering just a month ago and read many of those paper which use Mozilla's data. It's a very popular dataset in the field since it is both open and large. I'm sure Mike Hoye meant to criticize a part of that field, not academic CS research as a whole.

My impression of the field was there was a severe mismatch of skillset. The set of people with the scientific background to carry proper experiments, and the funding to do so, is very disjoint from the set of people who understand the field. That made a lot of the papers feel "off". Almost like reading text generated by a machine: individual sentences make perfect sense, the whole doesn't seem to go in a relevant direction.

As someone who's done a fair bit of practical software engineering, seeing academics study software engineers feels like seeing a WW2 veteran trying to understand how youngsters use snapchat. It feels very awkward for the youngster, just as it does for the software engineer. Which I imagine is one reason why Mike is pissed off.

There is some irony that businesses are much more scientific in this particular subfield than academia, because business incentives require the results to be reproducible and meaningful, over a longer period of time.

</comment><comment><USER19> The state of machine learning research these days seems pretty good. Essentially all research is published on ArXiv and there is a lot of code released too (though there could certainly be more).

I think openness has been a big contributor to the recent explosion in popularity and success of machine learning. When talking to academics about this, machine learning would be a great field to hold up as an example.

<comment><USER20> I'd say the opposite as a member of a group at my university who review ML papers. First off right now there seems to be a drive to explain many phenomena in ML in particular why neural networks are good at what they do. A large body of them reaches a point of basically "they are good at modeling functions that they are good at modeling". The other type of papers that you see, is researchers drinking the group theory kool-aid and trying to explain everything through that. At one point we got 4 papers from 4 different groups that tried to do exactly that. All of them are flawed, either in their mathematics or assumptions (that will most likely never be true, like assumptions of linearity and your data sey being on a manifold). Actually speaking of math, many papers try to use very high level mathematics (functional analysis with homotopy theory) to essentially hide their errors as nobody bothers to verify it.

<comment><USER21> >First off right now there seems to be a drive to explain many phenomena in ML in particular why neural networks are good at what they do. A large body of them reaches a point of basically "they are good at modeling functions that they are good at modeling".

Since this is closely related to my current research, yes, ML research is kind of crappy at this right now, and can scarcely even be considered to be trying to actually explain why certain methods work. Every ML paper or thesis I read nowadays just seems to discard any notion of doing good theory in favor of beefing up their empirical evaluation section and throwing deep convnets at everything.

I'd drone on more, but that would be telling you what's in my research, and it's not done yet!

</comment></comment></comment><comment><USER22> For a computer scientist, reproducibility means more work that they aren't paid to do. If I ask the Mozilla team to implement new feature X, the response will be either (a) point me to a donation link, or (b) We're open source, so why don't you implement the feature yourself? The computer scientist's response is the same.

<comment><USER16> Reproducibility for the computer scientist means including any code written and data collected or relied on in the scientific publication itself. In practice, getting there from here isn't literally zero work, since some actual human action is needed to bundle the code and data, but that effort ought to be negligible overall, especially if we make it a standard part of the scientific process.

<comment> semi-extrinsic Trust me, it's currently far from zero work to submit code with a research paper. I was recently the corresponding author on a software paper sent to a journal that at least verifies the code compiles and runs and produces expected output. Since the poor person testing the softwares submitted is permanently in the ninth circle of dependency hell, across all platforms and libraries imaginable, it took about fifteen emails back and forth plus an OS reinstall before everything checked out. And they said that wasn't anything extraordinary.

<comment><USER16> How about a platform centered around Linux containers (or maybe one of several OS containers or VM images), as the repository image?

I'm not saying the work is zero now, but maybe we can get there. If a researcher is developing on a platform where their repository is expressed as a container-like image, then they should be able to publish it for anyone to run exactly as-is. The container repo includes the data, the operating system, and any languages and libraries, with an init system that optionally builds the results.

<comment> semi-extrinsic Yes, I think we need to go in this direction. The problem is that the container system is yet another tool for researchers to learn. The first step is to get everyone using VCS and nightly testing. Many are still at the point of clumsily written, old Fortran code that gets emailed around and exists in N different variants. (Not that there is anything wrong with Fortran.) Many are at the point where if you email them a link to a git repo to clone, they're clueless about what to do.

<comment> Al-Khwarizmi It would help if Git didn't have such an awful learning curve (and I say this as a git user that already went through it).

I know researchers that used Subversion when it was on the rise, but they just abandoned version control altogether when Git became the generally preferred option.

</comment></comment></comment></comment></comment></comment><comment><USER23> We're working on this exact problem at Empirical [http://empiricalci.com](http://empiricalci.com)

We're building a way for CS researchers to run, share and compare experiments including their whole research environment.

We just applied to Apply HN and would appreciate your feedback: [https://news.ycombinator.com/item?id=11586948](https://news.ycombinator.com/item?id=11586948)

</comment><comment><USER24> > They publish LaTeX-templated Word docs as paywalled PDFs.

Somewhat tangential, but do CS academics actually write papers in Word? During my grad school days I did not encounter a _single_ paper 'typeset' in Word. Writing was usually done with a LaTeX and and a makefile in a git repo.

<comment><USER25> Just adding another data point: When I was in CS grad school (late 90s), everyone used LaTeX (including the journals). Then I started hanging out with neuroscience people, and everyone (including the journals) used Word+Endnote, with basically ad-hoc treatment of the figures.

<comment><USER24> How does Word work out when multiple collaborators are writing at once? One of the cool things about version controlling LaTeX is that conflicts, merges, etc are dealt with using standard tools, which is super helpful when you have collaborators across the globe furiously writing and redoing figures a few days before a deadline.

<comment><USER26> Well when you deal with merge conflicts in Word, your standard tool is Word. I don't think you can get much more standard than Office when you consider collaborators outside academia. It's not great for simultaneous editing (although I think this is now possible in 365).

It is, however, very good for tracking changes over versions. Many academics are not familiar with git, diff and so on and it's nice to easily see historical edits in the document. For simple documents like abstracts, it's much easier to send a Word document than it is to send a tex file and assume that everyone on consortium is going to be able to compile it (especially if you work with industry).

<comment><USER27>
        > although I think this is now possible in 365

It would want to have improved since I used it a couple years ago in Word 2013. The main problem was with citation managers - Word would give a paragraph lock to you whenever you edited a paragraph, and it would only unlock that paragraph after a save (either auto or manual). Of course, when you have a citation manager, they have the habit of changing all the paragraphs when you insert a new citation that changes the numbering (ie. [1] becomes [2], etc.).

</comment></comment></comment></comment></comment><comment><USER28> As a recovering CS academic, I have to say - boo-fucking-hoo. Tell me exactly where the funds to do this extra level of support is supposed to come from? Is Mozzila paying for those grad student hours? No? Well then.

<comment><USER12> We as taxpayers are already paying for your research.

Given that, it seems reasonable that you should make the modicum of effort to publish your code.

Also, don't construct a false dichotomy. Publishing your code (at all) does _not_ mean you have to make it production-quality or provide support for it.

<comment><USER29> > Publishing your code (at all) does not mean you have to make it production- > quality or provide support for it.

No. If you publish something that's incomplete or doesn't have all the right dependencies listed, etc, it's not really of any use. Writing up compiling instructions plus dependencies plus how to run it plus input files etc takes time and by the time you've got it to the state that someone else can run it, now it's "production-quality".

> Given that, it seems reasonable that you should make the modicum of effort > to publish your code.

There's currently no incentive/requirement to publish code, it uses time and does not increment your publication counter. Find the incentive and you'll start seeing published code.

<comment><USER12> > No. If you publish something that's incomplete or doesn't have all the right > dependencies listed, etc, it's not really of any use.

Again you're creating a false dichotomy which is just fundamentally false.

Even if you only published _half_ your code—to the point where it doesn't even compile—that's still helpful.

In the status quo, I have to write your entire code from scratch.

If you published what you have, I would merely have to debug the issues and figure out what the dependencies are.

I don't understand why it's so hard to understand that some code is better than no code.

<comment><USER29> > I guess researchers really are so incompetent with software that they think > code which requires debugging is useless.

So published papers are held to a high standard -- filtered through editors and peer review -- while publishing code can be half-assed at best? I still disagree; if it's worth doing, it's worth doing right.

</comment></comment></comment></comment></comment><comment><USER30> Reproducible is a major issue across science in general, but the difference is there's no reason why one shouldn't be able to easily re-run a defined analysis on a more recently updated data set to ask if conclusions drawn previously still hold. I actually published a side-project paper on this (in biological sciences) last year [1] - what was scary was there was such a lack of discussion surrounding this idea, despite the fact that large databases of biological data are CONSTANTLY changing and updating.

[1] Holehouse, A. S. & Naegle, K. M. Reproducible Analysis of Post- Translational Modifications in Proteomes-Application to Human Mutations. PLoS One 10, e0144692 (2015). ([http://journals.plos.org/plosone/article?id=10.1371/journal....](http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0144692))

<comment><USER31> The other difference is that as far as I know, computer science is the only discipline for which industry has solved the problem of reproducibility; it's one thing to be asked to design a method to run reproducible studies of humans, it's another to ask researchers to run `git remote add [https://github.com/user/repo](https://github.com/user/repo) && git push --set-upstream`. That's not asking for any support, or other effort on the researcher's part, and I frankly don't understand how the CS academic community doesn't have this as a standard when it'd be so easy to implement.

</comment></comment><comment><USER32> It is a real issue. The programming languages and software engineering communities have been working on evaluating software artifacts for a few years now:

[http://www.artifact-eval.org/](http://www.artifact-eval.org/)

This doesn't address open access, but it does make sure the software is usable by a non-author to reproduce the results of the paper.

<comment><USER33> @ open access: In my specific subfield of academic CS, a huge success story was ECOOP (one of the biggest conferences in the field) going open access last year. I'm hoping that there'll be other conferences following suit. My money is on open access being commonplace in 10-15 years.

<comment><USER34> Note that ECOOP went open access partly under the leadership of jpolitz's advisor. :-)

<comment><USER33> You had me confused for a second, I didn't realise I was talking to jpolitz :) Thanks for bringing that up. AFAICT, he (advisor) has done a lot to make that happen, indeed.

</comment></comment></comment></comment><comment><USER35> Help me convince people to do a few things a bit better: [http://da- data.blogspot.com/2016/04/stealing-googles-coding-...](http://da- data.blogspot.com/2016/04/stealing-googles-coding-practices-for.html)

(tl;dr of my linked blog post: Apply a slightly lighter weight form of some industry engineering practices in CS research coding. I think it's feasible. It doesn't solve all of the problems, because as discussed elsewhere in this thread, some of them are incentive-related and I'm not going to claim to have answers to everything. :)

(a) Convince more research groups to do their research on GitHub by default -- ideally, in open repositories. They get good hosted SCM, the world gets a better chance of seeing their code.

(b) Create more incentives, like the USENIX Community Award, for research that puts out its code. I'd say that in the systems community, a pretty decent chunk of the papers at SOSP, OSDI, and NSDI have code releases (of varying degrees of usability) accompanying them, though that's not a scientific count.

Mozilla could throw $1k to help create community-award-style incentives in the conferences they're interested in. Win-win. You get engaged with the community, you create some incentive for people to do the right thing, and you can use it as an onroad to deeper engagement with the winning authors (i.e., you can try to bring them in for internships. :).

</comment><comment><USER36> What it will take is creating a new system of research and development that ignores the traditional academic system. Because this has been a problem for a while and they clearly are not hearing the message.

The reason academic research works is because it takes risk on potential failures, because it's only donated or grant money anyways. But academic institutions fetishize academic papers, which is the problem from the article.

We need to legitimize research outside of the academic institution. Pay people to do research on their own time, if they perform it in an open-source, reproducible fashion. Incentivize it based on the reproducibility factor, but avoid attaching a profit motive.

Look at what Xerox PARC and Bell Labs were able to do before the penny- pinching bean counters took over.

Otherwise, it just sounds like "we want all this risky work done, and we don't want to pay for it."

<comment><USER25> Maybe I've drunk the kool-aid, but saying "academic institutions fetishize academic papers" seems wrong to me. It's like saying that developers fetishize working applications. It's not a fetish, it's the whole point of the thing. The output of research is research papers. Yes, sharing raw data and code are both good things, and should be promoted. But no one is going to take the time to look at either unless the paper presents good evidence for some novel result.

<comment><USER36> Papers don't do anything. Papers aren't executable. Papers are largely only useful in regards to writing other papers. To make the whole point of the exercise to be something that is inert and self-referential is the fetishization part.

<comment><USER37> Papers teach people things. You can't really hand someone some working code and expect them to iterate on it to produce something novel and insightful, because it won't be written in a high-enough language to express the core concepts flexibly.

<comment><USER36> I don't see how open source software is at all compatible with the view you've just expressed.

<comment><USER37> I don't see how that's related. There are loads of open source software projects that are maintained by only a single person precisely because they haven't bothered to properly document why it exists, what came before it, what important things it is demonstrating, or what needs to be done to improve it; the archetypical content of a research paper.

</comment></comment></comment></comment></comment></comment><comment><USER38> This is why I believe NixOS is so important. It allows one to completely freeze the entire development environment on one hand, but also does so in a declarative, well abstracted manner (vs VM image let's say) so tweaking/porting is actually feasible.

Until well get to a point where building/installing/administrating is not hours of bullshit, research (and free software) will suffer.

<comment><USER39> As an academic researcher I find it absolutely hilarious that you think the complex social problem of incentive structure and competition will be solved by some Unix OS.

If you are interested just take a look at the complexity of licensing/ownership of code written by a PhD student at a Research University in United States.

If you look at most of my Open Source code, I use AWS AMIs to share both data as well as OS + code, however I can do that only for side projects. The main thesis projects are typically very high value and consequences of sharing it far more complex to understand.

[https://github.com/akshayubhat/](https://github.com/akshayubhat/)

<comment><USER38> No that is not what I think at all, see my follow-up comments below. I just think the combination of shitty tools + incentive structure is even more insurmountable. This is a tough problem that should be attacked from as many fronts as possible.

> The main thesis projects are typically very high value and consequences of > sharing it far more complex to understand.

Commercial value, the university is just more stringent on the licensing/ownership restrictions, or something else?

<comment><USER39> There are several factors.

1\. Commercial value.

2\. Future grant applications (a competing group not sharing code will have better chance winning the grant.)

3\. Future of other students and collaborators in the group. If two PhD students write a paper, the junior student might wish to write extension papers without getting scooped.

And many more. Yet if a paper is important enough, independant researchers will often attempt at replication, this nowadays routinely happens in Machine Learning and Vision due to huge amount of interest. Also in several cases replication is fundamentally impossible, e.g. consider a machine learning paper that uses proprietary data from hospital attached to the university, etc.

<comment><USER16> I totally get that researchers' incentives are not aligned toward publishing it, so no need to explain that further. There are costs and downsides and probably not enough benefit to them. That's fine. Everyone works within their system of incentives.

If it's paid for by public dollars, then the code and data belong in the public domain eventually. I understand there are exceptions like hospital data affected by patient confidentiality - that's fine. However the code released by that researcher should be capable of reproducing their results with that data set plugged in (such as by someone else who has access to it).

As a taxpayer, my concern for publicly funded research is maximizing benefit to the public good. I understand your point about follow-on research, and I'm not saying that I'd expect the code and data to be made available immediately with publication, but that deserves to be the case some reasonable time afterward (like a year). I understand that researchers' incentives are not necessarily aligned toward making it public; I am saying that people who fund research (including taxpayers through the political process) should require and expect it. Keeping it private indefinitely is a degree of self- centeredness that does not strike an appropriate balance between benefit to the researcher and to the public in my opinion.

<comment><USER39> I never understood the meme about "public funding" translating into "public domain". Just because research is "publicly funded", does not means that the "public" owns it or even has a right to ownership. Public education is publicly funded does not means that government can ask for every drawing drawn by 9 year old in classroom to be in the public domain :) . In fact its actually opposite ([https://en.wikipedia.org/wiki/Bayh%E2%80%93Dole_Act](https://en.wikipedia.org/wiki/Bayh%E2%80%93Dole_Act)), given that Universities can and do patent inventions from publicly funded research.

Further funding arrangements themselves are very complex, a professor typically procures funding from University, NSF, NIH, private companies, donors etc. In such cases if NSF adopts a hard line approach that any research touching its dollars ought to release code under say GPL, it would make it impossible to collaborate. Finally all requirements aside, one can always release intentionally poorly written code in form of MatLab .m and compiled mex files. I have observed several such cases, where the code can demonstrate a concept but is intentionally crippled.

Finally graduate students, graduate and are paid for doing research which is publishing/presenting papers at peer reviewed conferences and journals. If what funding agencies really seek is ready made software they ought to fund / pay at the same level as software developers (as many companies do).

<comment><USER16> > Just because research is "publicly funded", does not means that the "public" > owns it or even has a right to ownership.

I didn't make the argument that the public owns it or has a right to ownership, though I suppose that some people might and so I can see why you would touch on that point.

I would describe my view as like this: public funding is subject to the political process, and voting by taxpayers (directly or indirectly through voting of politicians or their appointees). As a taxpayer, I prefer to make public domain publication a requirement of publicly funded research, and I think every taxpayer should too. I consider the goal of public funding of science to be the benefit of public good, and believe that public good will best be served by public domain publication of all data, code, and analysis methods. (Whew, there's a lot of "pub" and "public" in there!)

One might reach my position by working backwards from, "Why do we as taxpayers agree to fund science with government money?" It's certainly not to give researchers prestige or jobs! (Those may be necessary parts of achieving public good, but they're not the goal which is the public good, and if they're in tension with public good then the public good probably needs to win.)

I don't seek ready made software; not at all. I only seek adequate disclosure of data and analysis methods sufficient for others to easily verify it and build on it. See for example the attempt at replication in [http://www.peri.umass.edu/236/hash/31e2ff374b6377b2ddec04dea...](http://www.peri.umass.edu/236/hash/31e2ff374b6377b2ddec04deaa6388b1/publication/566/)

> In such cases if NSF adopts a hard line approach that any research touching > its dollars ought to release code under say GPL, it would make it impossible > to collaborate.

I will need to think more about this issue. I might be willing to accept the downside as a taxpayer. I'm not sure I understand it well enough what the friction would be to collaboration at the moment. If you're referring to the GPL specifically, then yes I agree that's probably the wrong license - <USER2> domain would be more appropriate.

I would be OK if this was simply an electronic log of the data as well as all machine commands that have been run on it - something that is recorded automatically by the operating environment. I am truly not looking for "working production code". But, those sequence of commands should be reproducible if someone "replays" them; a verifiable digital journal. Publishing an article that's difficult-to-reproduce feels like producing the least possible public good while still getting credit. Publishing an article that's fully and automatically reproducible, because it contains references to all of the data and code that yield the results as an executable virtual machine with source code, provides the maximum public good, and that's what I want science funded with public money (and ultimately all science) to work toward. (I realize that this is just like, my opinion man :)

<comment><USER39> You are correct in expecting return of public investment. Actually NIH has a policy that explicitly favors "Basic Scientific Research" over applied or application of research. According to NSF and NIH, the primary goal of government funded research is advancement of science, this is done via conducting experiments, and publishing results at peer reviewed venues. The peer review both during grant application and at publication stage factors heavily into assessment by funding agencies. If tomorrow NSF were to give significant weight to availability of source code (They actually do that to a small extent), it might set up perverse incentives. A small percentage of federally funded research goes into Computer Science and even a smaller fraction involves results where there is enough demand for software. Another aspect of academic funding that people don't get is that research grants unlike say contracts have a significantly different set of expectations associated with them. E.g. a student can get NSF Fellowship, claiming he want to cure diseases using machine learning, only to later spend 3 years working on music recommendation system (True Story!).

Regarding the economics study you linked to, I am very much familiar with that study having seen the interview of graduate student on Colbert Report. For non-CS fields the quality of code is anyway so bad that its much more difficult. Further several researchers rely on proprietary tools, which only make this task difficult.

In my opinion the correct way is not by having NSF impose rules, but rather by having venues that accept papers (Conference & Journals) insist on providing software. However this is easier said than done, since its a competitive two sided market.

Regarding actual licensing issues, I can assure you that GPL is second favorite license of choice favored by University IP departments, the first one being "All rights reserved with modification explicitly forbidden, except for reproduction of experiments."

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER40> Funny, as students we were expected to solve the exam problem and "show your work"...

I think the root a the problem is that the goals of the researchers are not aligned with the goals of Science. This isn't a criticism of the researchers but instead of the "game" they are forced to play.

For example,the goal of Science is to move the ball(knowledge) down the field for the benefit of mankind. We don't reward researchers for doing that, at least not very well. We reward researchers for writing _papers_ full stop - not for making their research easy to reproduce or build on.

</comment><comment><USER41> Its not just the researchers' fault. Maybe the industry should help. Mozilla is a major stakeholder of the web platform that makes distribution easy. Lets make sure web is the best platform to do all the research.

* Provide great scientific and matrix manipulation libraries within the browser. WebAssembly isn't going to solve this. Why would the academia rewrite everything?

* Provide tools that help research being open. Uploading your code to Github isn't a solution. The real solution is making it easy to use and link other person't research. Can we make research as accessible as a javascript file that you include in your html file to run. And it shouldn't cost the creator to host/maintain it. Offline web sucks(still) and it costs money to host your servers.

* Provide incentive to use the web for everything. A great one would be an easy to use and debug toolset and an easy set of methods to get data in and out, an editing environment that can be setup with one click. The closest is iPython Notebook. And it takes work to get there.

Sharing should be default and easy. If it isn't we are no one to complain.

<comment><USER42> >Uploading your code to Github isn't a solution

Why not? This sounds like exactly the solution. It's no burden on the researcher - they don't have to alter their research methods to fit a new system, they just dump the code, and leave it up to other users to reimplement.

Building web tools to allow research sounds like precisely the wrong solution, at least in the short to medium term. Research funding doesn't go far enough as it is, you're not going to get researchers changing their processes entirely for no gain. What if they need custom hardware, or access to tools and libraries that haven't been implemented?

Sharing _is_ default and easy. That's why Github has exploded.

<comment><USER43> You forgot to consider IP issues. Schools vary a lot on their policies by but for many the code belongs to the school and can not be open sourced without permission which requires extra work. Funding sources have their own IP deals to consider too.

It all sounds so easy until you look at the actual constraints. Professors are usually smart and experienced and they have thought about this stuff a lot. If it was as easy as you thought, it wouldn't be a problem.

I publish all code as a matter of lab policy. I chose where to set up my lab partly so that I was able to do this. Not everyone has this luxury or makes this a priority.

<comment><USER42> You're right, I did, but I wasn't actually addressing ease in some absolute sense, I was talking about the appropriate tools for the job.

If researchers have the legal right to publish their work, I can't see any reason why github wouldn't be exactly the place they'd share it, rather than some custom online research system as proposed by the parent.

That said, I don't have any experience in CS research, it's not my field, so I may be wrong about that, do tell if so.

<comment><USER43> We share our code on github and publish the commit hash for the code that generated the results in the paper. That way we can continue to develop the code after publication but readers can retrieve the exact code described in the paper if they wish. Simple and effective.

But again, the IP rules at my university allow this at my sole discretion, which is unusual.

</comment></comment></comment></comment></comment><comment><USER44> Jean Yang @ CMU (together with Sam Tobin-Hochstadt @ Indiana) posted a response to this on her website:

[http://jxyzabc.blogspot.com/2016/05/myth-cs-researchers- dont...](http://jxyzabc.blogspot.com/2016/05/myth-cs-researchers-dont-publish- code.html)

Releasing code is a widespread practice in the programming languages and software engineering communities, and one that is getting stronger (see [http://artifact-eval.org](http://artifact-eval.org)).

If you are a CS researcher, please fill out this survey of open source and data in computer science:

[https://docs.google.com/forms/d/1U6I3B2XAqGJDYq101e14PS78jhG...](https://docs.google.com/forms/d/1U6I3B2XAqGJDYq101e14PS78jhGl4pTHy6eDx2zx0j4/formResponse)

</comment><comment><USER45> I guess I can see why many researchers are not inclined to publish their code. I've worked as a research assistant in the institute of microelectronics of my university for a year and the code quality was somewhere between mediocre and downright terrible. And that's not to mention the absence of sane software engineering standards like bug tracking or code reviews.

</comment><comment><USER46> From reading this discussion, it seems that IP rules are inconsistent, and therefore part of the solution needs to be to perhaps have funding agencies enforce a standard license. With that barrier eliminated, scientists wouldn't expend valuable time negotiating with their school's layers.

</comment><comment><USER47> Personally, I'd feel code with light commenting would be better suited for getting a "practicable" point across to readers then a paper will.

You won't need to put so much weight on describing some of your practices, you can just how how your solution is better.

</comment><comment><USER48> > But that never happens. Because CS researchers don't publish code or data. > They publish LaTeX-templated Word docs as paywalled PDFs.

Correct

Publishing in pseudocode made sense some 30y. ago

Today if you're publishing in pseudocode I personally think it's someone who is doing that because can't write a Hello World in C

Oh and also let's stop the "research comes only from universities" idea

</comment><comment><USER49> Fund the research you want to see in the world. Incentivize it as you think is right. Don't whine.

</comment><comment><USER50> Codalab is designed to solve _exactly_ this problem: [https://worksheets.codalab.org/worksheets/0x818930127c4d47de...](https://worksheets.codalab.org/worksheets/0x818930127c4d47de84c1ceaadf04d014/).

</comment><comment><USER51> making research reproducible with publications that actually contain their code (in addition to prose and graphs, all connected together) is one of the goals of Beaker Notebook, which has a publication server where you can share such results, and then with one click open them up in the cloud, modify them as you see fit, and then run them again (if you have seen Beaker before, this last part, the cloud hosted version, is new as of last week).

[http://beakernotebook.com](http://beakernotebook.com) and [https://lab.beakernotebook.com/#/publications/featured](https://lab.beakernotebook.com/#/publications/featured)

<comment><USER52> Wow, looks impressive and that it comes from hedge fund is cool too. It's like Jupyter but much more robust, multiple languages in single notebook and ability to share data across languages out of box is really nice.

</comment></comment><comment><USER53> I would not be surprised if most academic CS research is not reproducible. This is true for many other fields outside of CS, and I've seen it first hand in machine learning. It's a problem but it's also just how things are.

</comment><comment><USER54> There are other issues right now, like impartiality and bought research, like Microsoft at Cambridge University.

I recall the professors at our uni treating the undergrads with contempt as wasting their precious lucrative research time.

<comment><USER55> Precious? Yes. Lucrative? Hardly.

</comment></comment><comment><USER56> Maybe it can be thought of as a tooling problem? Say, a plugin that allows a one-click publish code + data from Matlab, and then it all goes up on a well- indexed page so that others can download/run it.

<comment><USER57> I doubt the problem is that academic CS researchers don't know how to publish their code, but rather that the disincentives are usually stronger than the incentives.

<comment><USER58> Is there even code to publish? I am under the impression a lot of papers from Bugzilla data are of the form "we imported the data into Excel and had a hand- crafted one-off spreadsheet".

<comment><USER16> In that case, yes: the spreadsheet itself consists of data and analysis over data (aggregations over columns and rows, etc.) so the spreadsheet itself would ideally be version controlled and published.

The idea isn't to ask researchers to formalize what they make more than before, but to include fully reproducible details in the publication. A spreadsheet is totally fine because you can see how it works, reproduce the result, and tweak the inputs/methods to build on it.

</comment></comment></comment></comment><comment><USER59> It seems like politely writing to the researchers and asking if they still have the code lying around might have good results. (If nothing else, it lets them know someone cares.)

<comment><USER60> A colleague was working on a replication study. We got the code from the original researcher and another researcher who did a follow-on study. The code barely runs, and the results seem off. I spent days debugging to no avail. Just because researchers provide code does not mean it is well-written code, or that it necessarily works.

<comment><USER16> Then I'd be skeptical about those results. Releasing the code allows others to judge the likely accuracy and integrity of the results. A lot of things can go wrong in complex, multi-step computational processes. If care and rigor has not been put into them, then I'd have no reason to believe in the integrity of the output. I want the general public right to judge that, as well as build on it when it's useful and valuable.

Every publication involving data and technical analysis should publish them to a degree that makes validation possible in at least as detailed a way as portrayed in "Does High Public Debt Consistently Stifle Economic Growth? A Critique of Reinhart and Rogo ff" ([http://www.peri.umass.edu/236/hash/31e2ff374b6377b2ddec04dea...](http://www.peri.umass.edu/236/hash/31e2ff374b6377b2ddec04deaa6388b1/publication/566/))

</comment></comment></comment><comment><USER61> What about this promising recent link? [https://gym.openai.com](https://gym.openai.com)

</comment><comment><USER62> I really wonder, what's the difference between computer science and mathematics?

It seems like CS is a subset of math.

<comment><USER63> In a word - specificity. The same question arises in a number of older fields of study, for example "is chemistry the same as physics?"

<comment><USER64> Here's a guess: the short answer is that the physicist would say 'yes', but would be unable to solve problems that the chemist would consider trivial.

</comment></comment></comment><comment><USER65> I would extrapolate that to all of academia. If you want to work on something that's useful, insightful and will make the world a better place, the ivory tower of academia is great if you want to live in denial in an echo chamber.

<comment> rudi-c That's a strong claim. While I'm sure the sentiment is understandable for some fields of academia, others have produced significant results that have been adopted by the industry, sometimes very quickly.

Machine Learning has a lot of theoretical results produced by academia, but the more practical techniques (decision trees, SVMs, neural networks, etc) also all came from academia. The engineers scaled the algorithms to run on bigger datasets, but the initial work is still driving those systems.

Graphics research has seen comparable contributions between industrial research labs and academia. Sure, a higher proportion of papers from academia don't end up practical, but the number of papers that are very practical makes that quite irrelevant. It's to be expected since you can't predict what works in advance, and industrial research labs just don't bother publishing negative results, not that they don't get any.

Many programming languages and compiler techniques came from papers.

I could go on but you see the point - it depends on the field.

<comment><USER65> But that doesn't say anything about the person in academia who writes a speculative generic research paper about a way too simple implementation of a decision tree. I don't want to demean that accomplishment, but for me it would be hard to have sustained excitement from that. Also I think that a counter argument is that just because the initial work of X, Y and Z algorithm were sketched out in academia, they still very much so would have been scaled in industry once the actual need has been found for them.

</comment></comment></comment><comment><USER66> What percentage of the CS research based on any data at all? It is mostly purely theoretical. Still, it would be nice if they published their proofs as a Coq code.

<comment><USER67> I think you're conflating a very narrow slice of CS (theoretical computer science) with the larger field. There's a huge amount of CS research that relies on gathering and analyzing data, building systems, etc. Theoretical computer science is actually a very small slice of the research pie.

<comment><USER68> At least in programming languages, systems, and formal verification project code is both available and often evaluated along side the publication. For example FSCQ([https://github.com/mit-pdos/fscq-impl](https://github.com/mit- pdos/fscq-impl)) from MIT, Verdi ([https://github.com/uwplse/verdi](https://github.com/uwplse/verdi)) from UW, Dune([http://dune.scs.stanford.edu/](http://dune.scs.stanford.edu/)) from Stanford, and many more.

These are just the areas I work in, its very hard to making sweeping statement about CS as a full field since it is very diverse sub-community to sub- community.</comment></comment></comment>