Science Is Getting Less Bang for Its Buck - <AUTHOR> https://www.theatlantic.com/science/archive/2018/11/diminishing-returns-science/575665/ <comment><USER4> This seems like the obvious result of many of the unhealthy forces against science today: replacing professorship with part-timers, rewarding publishing quantity over quality, pressure to promise a quick monetizable result over in- depth basic research.

I don't buy the "low-handing fruit" argument. I think it vastly overstates the fraction of scientific knowledge that we possess. Contemporary scientific research is in a state of crisis: unhealthy and dysfunctional. Until we pay the most talented researchers to spend decades on in-depth study instead of spending pennies on segmented, p-hacked, post-hoc conclusion garbage papers, major discoveries will continue to disappear.

<comment><USER5> > Until we pay the most talented researchers

It's not even that. One of the contributing factors is that we are no longer hiring for talent or merit. If you're a male child interested in science and in high school right now you've been through 8 years telling you that you should not go into science because they need more women of colour who follow the right religions and have the right sexual orientation. You're constantly bombarded with "SCIENCE DOES NOT WANT MEN".

And sure, we can say "well if he had the necessary grit, this would not discourage him". No. No amount of grit will get you in when governments are shaping science funding to punish having male staff. No amount of grit will get you in when indoctrination training is mandatory for the selection committees. No amount of grit will get you in when the first filter in a hiring process is to remove all males applicants.

I'm sorry but science has to go through a decline in the West for a bit. We still have some remnants to be worked through the system, but the next couple generations of Western scientists are going to be generally pretty low quality. On the bright side, science is not exclusively a Western tradition at this point. Other cultures that at least pretend to still focus on merit and talent will carry the scientific tradition forward.

<comment><USER6> > If you're a male child interested in science and in high school right now > you've been through 8 years telling you that you should not go into science > because they need more women of colour who follow the right religions and > have the right sexual orientation. You're constantly bombarded with "SCIENCE > DOES NOT WANT MEN".

Do you have any examples of teens / young adults being discouraged from going into science because they are white and male? There are certainly some (I'd say good!) programs to encourage women and minorities, but that's quite a different thing from men being actively discouraged.

Scientific fields are still significantly more male than female, even among recent graduates.

<comment><USER7> > Scientific fields are still significantly more male than female, even among > recent graduates.

Why "still"? Majority of mans are genetically selected to be adventurers, while minority of woman are adventurers. It's impossible to change genetics with advertising, so we will never have 50/50% split, until human genome will change. Why not just accept natural distribution?

<comment><USER8> I'm not convinced that joining the scientific community is an adventure or that this genetic difference exists. The differences can be explained without using genetics.

For example, there is a popularly held opinion that girls are bad at math. Girls who believe they are bad at math might try less in math courses. Girls that receive lower grades in math from trying less might be intimidated by the math requirements of a science program.

<comment><USER9> It's not an adventure, but it is a high-risk with a slim chance of a very high payoff endeavor, and men are typically more drawn to that than women, for example there his a huge gender disparity in fishing the Alaskan fisheries or base jumping.

<comment><USER7> Offtopic: Can you explain to non-native English speaker, please, why "adventure" does not match "a high-risk with a slim chance of a very high payoff endeavor"? I saw "adventure in science" few times, so I assumed that someone who does this "adventure in science" can be named "adventurer". What is wrong in my conclusion?

<comment><USER8> I wouldn't consider a career in science to be high risk. It's often easy to transfer from academic science to private engineering. Science degrees holders have respectable earnings on average.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER10> I'm not sure where to begin on this, so I'll resort to a brain dump.

The Nobel prize has become steadily more political. More senior researchers in the field have had more time to accrue political power, so expect the awards to reach further back in time.

Research productivity has dropped, though. Today a professor in biology has a more than full time job just to get money for their lab. So you go through gradschool, go through postdoc, and when you've got all this training, you stop doing science and become a grant writer. If you don't bring in money, you don't have a lab.

For the postdocs and grad students, they live very insecurely. Their pay is poor, they live knowing they will probably relocate somewhere entirely different in three or four years, so there's no point having links into the local community. They work for someone has no training in management and has no time to deal with them if the money is going to flow. And there are still lots of places where the postdocs aren't considered full time employees of the university and don't have, say, health insurance. These are hardly the conditions that you can expect to produce good work.

Fields are mined down to the last details rather than looking at unexplored areas nearby. There are many reasons for this. As a grad student you work on what your advisor works on. As a postdoc, you work on what your lab works on. As a professor, you work on what you worked on as a postdoc and it requires multiple years and lots of luck to reorient a lab. You need a community to provide evidence of your competence in order to advance at each stage and get tenure, which you can't easily get if you wander out of an established field. Nor is there any training on how to look effectively find things to work on.

<comment><USER11> Is this only a US and European problem? when I was in college in the oughts their was a popular trend in Physics PhDs to move to china following there Grad program as China provided a block grant to them to start a new lab.

The pattern whereby professors have to apply for small grants continuously to keep a lab funded and operated is an inordinate waste of time vs. a block grant with periodic evaluation.

<comment><USER9> Keep in mind by small grant we mean 100-200k towards a million.

<comment><USER10> Which _is_ a small grant. You would be hard pressed to run a five year software project on a million dollars, much less a project that requires real laboratory overhead and materials.

</comment></comment></comment></comment><comment><USER12> Quickly scanning comments I do not think anyone else has brought up: administrative bloat.

More money is being spent on science, but is more money actually making it through the administrative bloat encumbering most institutions to the actual performance of research?

Anecdotally, I have a colleague who has received funding from the NSF and the amount of regulations and paperwork and various travel and meeting-related obligations related to the funding soak up so much of the actual dollar amount supplied. (You have to use your funding dollars to satisfy the various required meetings, travel, and paperwork-filling.) The constraints are so ridiculous that satisfying them consumes nearly all the resources the NSF provided, and the little that remains is actually not sufficient to perform the research with. Worse, he has now wasted months of his time satisfying various oversight requirements administrated by both the NSF and the research institution he works in, leaving him an unreasonably small amount of time to actually achieve any significant progress on his work. Once this round of funding dries up, he will be left with no choice but to repeat the process in order to secure some more funding to continue to barely make progress on his stated research goal.

If I had to make up a number to describe the dollar efficiency of research funding, in some cases I might assert it is negative: Not only is it just being soaked up by self-serving, efficiency-draining administrative requirements, it literally destroys the most valuable resource (time!), leaving the researcher with none to actually engage in their subject matter of expertise.

<comment><USER13> it's huge. Similar in an european ERC-funded lab: the PI is constantly traveling, there is little oversight of the work let alone actual scientific output. It feels like a large portion of the funding is designed to keep a lot of people people busy doing nothing.

<comment><USER14> At least the US government treats it like a "jobs program", just like everything else.

</comment></comment></comment><comment><USER15> In my opinion science is getting less bang for bucks because the low hanging fruit has been picked by now. The big fundamental structures describing nature its working are more or less known. It's about details nowadays and it takes more time and effort to get the details right. You can see that it takes multi disciplinary teams nowadays to discover the connections between these large systems and how they (how all in the universe) connect. People are curious which is our most precious gift from nature and science will go on as long as people stay curious. But yes, not all science is about returns of value in economic sense, a lot of it is fundamental and also often results of science is laying in a drawer waiting to either getting monitized or used for further discovery. Science moves by spurts and hiccups, not straight forward with an even pace.

<comment><USER16> > The big fundamental structures describing nature its working are more or > less known. It's about details nowadays and it takes more time and effort to > get the details right.

This (bullshit) is what they thought in the 1890s,and is why physics was considered boring, basically a dead field. Oops!

<comment><USER17> Difference is, this time around we might be standing at the edge of what we can possibly know and/or understand.

<comment><USER18> And how could we possibly estimate if we are or not?

</comment></comment></comment></comment><comment><USER19> That observation is decades too late. Big companies set up sizable research labs in the 20th century. After WWII, it was expected that a big industrial company would have a sizable R&D operation.

Then, in the 1980s, those big R&D operations stopped paying off. Gradually, the big corporate labs closed - Bell Labs, RCA's Sarnoff Labs, Xerox PARC, Westinghouse Labs - gone. IBM's labs are far smaller than they once were. The payoff wasn't there. The easy hits from research were gone.

<comment><USER20> Hum...

My reading is that R&D facilities couldn't compete with the sheer amount of small improvements that came from the uneducated labor force. Japan grew out empowering the employees to do development, and the entire world copied it. What leaded us on a path where people are making the Uber-for-X apps and some actually getting rich that way.

And that is exactly the opposite of a sarcity of low hanging fruit.

</comment></comment><comment><USER21> Another problem with this data is the rise of team science and huge author lists for papers - something the Nobel prize isn't really able to deal with. Think of the human genome paper or the Higgs boson paper - both good 21st century science but lacking the easily identifiable "super scientists" the Nobel committee is looking for.

Additionally it is hard to fully evaluate the impact of scientific work fully until many years have passed since publication.

It's no wonder they prefer to find their winners in the 80s. ..

</comment><comment><USER22> A lot of 'money spent in science' has not really been spent. Just offered with lots of requisites that nobody can reach and then recycled for other projects. If granted, the money can be blocked for months (instead to pay an entire year to scientists, you pay 7 months... for one year of work), and can be partially siphoned off again using bureaucracy. Some politicians are very fond of this 'miracle of bread and fishes' trick.

"In the last ten years we spent '10 millions' in science projects". Looks great.

"In the last ten years we flashed the same million 10 times before to put it again in our pocket". Not so great, often the real thing.

</comment><comment><USER23> I actually don't see this from his data. It seems more like he started with a hypothesis, wrote the article as if he was correct, and then gathered data to try to support it. Then after seeing the data wasn't as commensurate with his idea, he wrote around the inconveniences.

<comment><USER24> Yep. I don't know what article the other people in this thread read, but it wasn't the same one I did. At best, the graphs can conclude that scientist's opinion of nobel prize winners stays about the same, regardless of time.

</comment></comment><comment><USER25> This is why science should be led by the government, not for profit companies. Science is for the greater good of humanity, it should not be solely about making money.

<comment><USER26> I love science (at least in the ideal form). That being said, if knowledge provides a real, tangible benefit to mankind, people should be willing to pay for it. If it doesn't, people shouldn't be forced to pay for it.

I agree that some valuable research would go undone if performed by companies because the time window to see a return on investment would be too long. Unfortunately, academia is a huge mess, and we really need a new approach to basic science.

<comment><USER27> What's your definition of basic science?

The requirement for Science to produce real, tangible benefits to mankind for it to get funded makes a lot of it unfeasible, because a good bit of it is useless on it's own, but is good to figure out because of potential uses in the future.

<comment><USER28> The real problem is we won't know if something is useful until after we get results.

Gravity is a perfect example - physics has a lot of open questions on gravity. What if someone closed one of them? It could mean nothing, just a slightly better understanding of why things fall - or it could be a major change that allows us to create "anti-gravity paint" making a self-supporting colony on Jupiter simple (image all the "land" you could own there - whatever "owning land" on gas planet means)

<comment><USER29> Come on, such a thing would allow for reasonable space travel even better than for just making colonies. It is essentially Mass Effect Element Zero - <USER2> could create gravity wells for things to "fall into", creating enormous accelerations of space ships while at the same time protecting the crew from the effects.

Potentially even a kind of faster than light travel.

As a side boon new kinds of materials can be created in extreme artificial gravity conditions some this maps to extreme and impossible to achieve otherwise pressures. Including just creating stars in a controlled environment making for immense energy availability. Instead of magnetic fusion containment you'd have gravitational fusion containment.

This is how powerful practically breaking gravity can be.

<comment><USER28> I just hand wave assumed a slightly different set of rules. With your set of rules (which is at least as likely as mine) you are correct. Since both of us are speculating we are both equally right. Of course it might be that better understanding of gravity proves that artificial gravity is impossible.

</comment></comment></comment></comment></comment></comment><comment><USER29> I'd personally bet on some sort of major breakthrough in space travel or energy sources or both (perhaps due to new physics) to have comparable results to the decades of the atom.

Being that, there is just not enough gain or even potential gain of technologies being surveyed right now, even considering full on genetic engineering and universal nanoconstructors.

Well, unless immortality or nigh immortality of some form happens. Or we found a way to expand our intelligence and performance in some huge way. And then it becomes available.

Compared to atomic and subatomic physics AI is a joke and genetics is at witch doctoring levels. Self driving cars are like the steam experiments in 1800s at best, with even less of an impact.

At least genetics has potential to change everything, more than automated transportation ever can.

Nanotechnology is making some progress but not nearly fast, good or cheap enough.

Another breakthrough could come from side physics like photonics.

</comment><comment><USER30> This could also be titled: "the foundational truths in a field are more consequential than subsequent findings".

</comment><comment><USER31> What if the survey respondents value earlier discoveries more because they understand them better? I remember my mathematics professors saying, once they chose a specialty within a field, there wee only a dozen or so people in the world who could discuss it on the same level. So concurrent appreciation is rare within a single field, let alone across fields.

But, as a high schooler, I was taught the basics of Einstein's general theory of relativity. I couldn't do the math, but I appreciated its value. Maybe the respondents undervalue recent discoveries in different fields.

<comment><USER32> Effects like that could only be measured if they repeated the same survey many times over decades before publishing. Even then it would be confounded by a lot of other factors.

Instead they've marched forward using junk science to discredit science. I found it a really weird and off key article.

They might be right but I wouldn't use this article to demonstrate it. (And I otherwise have a lot of respect for the authors.)

</comment></comment><comment><USER33> A huge part of this is the general perversion of science. It starts at the bottom with the abuse of p-hacking by lowly grad students. It goes all the way to the top with fraudulent for-profit science by Big Pharma to get drugs and medical devices approved that don't work. All of it is part of the academic- medical industrial complex that views science not as process for discovering truth, but as a tool that can be used to generate profit.

There are enormous breakthroughs that I know are very close, but simply inconceivable due to the academic-medical establishment that has stranglehold on science.

</comment><comment><USER34> Ex-professor here. Reason for this is trivially obvious to any outsider observing the day of an academic: A dazzling torrent of time-wasting grant applications, awards and other such nonsense.

The system is basically designed to take productive scientists, and waste them as quickly as possible. After all, how better to protect your mediocrity?

Never been happier than after leaving...

</comment><comment><USER35> Does anyone else find it fascinating the Patrick Collison (the co-founder and current CEO of Stripe) co-wrote this article?

The dude makes me feel inadequate like no other.

</comment><comment><USER36> Probably impossible to find out, but seeing a comparison - however loose - <USER3> other countries would be ideal.

Annecdotally, it seems there's at least one significant leap per week listed on HN that can be attributed to China. Reaching 100 million degrees from a fusion reaction (from earlier this week) comes to mind.

But of course the general public isn't enlightened by the mainstream media about such things.

<comment><USER37> The Chinese fusion reactor was not a leap. Already achieved in JET (Joint European Torus).

That said, if China decides to pour a lot more into R&D, it'll be doing leaps in no time, simply due to brute force.

<comment><USER36> Link to JET achievement? Please??

Chinese? European? Any comparison would be better than none.

Brute force? Does it matter? Results are results, yes?

The arc of my point is, the USA so often as a very self-serving, often myth- based view of itself. It's as if no one else in the world might have a better approach. That's a mistake.

<comment><USER37> JET achieved 0.7 Q, 24MW in 16MW out.

It operates somewhere around 100-200M Kelvin. [https://www.scienceinschool.org/2013/issue26/fusion](https://www.scienceinschool.org/2013/issue26/fusion)

> Brute force? Does it matter?

No, I doesn't mean to say it as a negative thing. Fusion research funding is very low compared to what is needed to build big enough devices. Because when it comes to fusion, size matters, as efficiency goes up with size.

And that's what I meant by brute force. China can simply build a bigger one and reap the benefits of size.

> Results are results, yes?

Yes, and more data is always better in plasma science.

</comment></comment></comment></comment><comment><USER38> Why do they assume something went wrong? At some point, we ill reach a point where there is far less left to discover or invent. (other than AI -> as long as anyone has a job, there's always room for automation, yikes!)

We assume that technological progress can continue forever and ever. I think this is an incorrect presumption.

</comment><comment><USER39> Excellent book on this topic.

Big Science: Ernest Lawrence and the Invention that Launched the Military- Industrial Complex

[https://www.amazon.com/Big-Science-Lawrence-Invention- Milita...](https://www.amazon.com/Big-Science-Lawrence-Invention-Military- Industrial/dp/1451675763)

</comment><comment><USER40> Considering how large the corporate sector is today and how corporate inspired the government sector is today... Science is remarkably unaffected.

We have tenure and sociology departments, a teacher-researcher combo tradition... All stuff that does not lend to ROI calculations or the kinds of resource allocation done elsewhere.

</comment><comment><USER41> I'll agree with Yaa101 that a big part of the story is the picking of low- hanging fruit, or to put it less metaphorically, that in the first 100 years of institutionalized science (let's say 1850-1950 without getting caught up in the exact dates), there were a lot of fundamental questions that could be addressed through the application of relatively systematic, rigorous observation and experimentation, and modeling with the kind of math you can do on a chalkboard.

Within this time period, though, a lot of these questions were addressed and the _new_ questions that arose required more data, better instrumentation, and more advanced mathematical modeling techniques to address.

In my own field (geology, in particular tectonics and earthquake studies), this was laid out in a very explicit manner: the fundamental mode of observation is geologic mapping, and the terrestrial surface of the earth slowly got mapped. The mapping of the past may be refined or re-interpreted but rarely does it need to be redone from scratch. It is done to a reasonable level of resolution. There is still a lot of unknown under the ocean basins, but we have strong theoretical and empirical arguments for why those areas are not as complex as continents and therefore less interesting.

The late 1960s through the mid 1980s saw the development of plate tectonic theory which _completely_ revolutionized the science. Now, 50 years in, we have some second- or third-order questions but most of the first-order questions have been addressed.

Today, the major developments of the field come from better instrumentation, for the most part. In the sub-field of tectonics, progress comes from the development and application of new methods for dating rocks or other geologic features (including things like exposure dating or 'how long as this rock been at the surface of the earth'), and from using satellite-based measurements of earth deformation (GPS and radar interferometry) to actually measure the motion of tectonic plates and sub-plates. Additional, continuous refinements in seismic imaging of the subsurface (driven by the oil industry primarily) has also been very helpful.

This stuff is really expensive! It's hard to go camp out, hike a bit, make some observations, and write a good paper. The instrumentation to get the age of rocks might cost $100,000 and then when consumables and salary are factored in it might cost $500-$2000 per sample. You might need 10-20 samples to really find out anything new in your 10-km by 10-km area of interest. And I believe that geology is quite cheap relative to high-energy physics or whatever. Major geophysical experiments can cost millions. We piggyback on physics and other tech to a large extent---launching GPS satellites for example, or using obsolete particle accelerators for geochemical measurements. The oil industry spends (tens of?) billions a year acquiring data as well but very little of it becomes public or available to researchers, though the cumulative data release from industry is significant. [NB, I may be 1 order of magnitude short on any of these numbers.]

In general there have been very few major theoretical advancements in the past 20-30 years. We have gotten better at recognizing coupling between tectonic process and earth surface processes, and as instrumental datasets slowly increase (as we observe more earthquakes, etc.) some smaller boxes get checked. However, a geologist from the mid-1980s would be able to navigate today's scientific landscape pretty well. The fads are different but like any fashion, many are cyclical and were fads in the 80s too.

I personally see advancement coming from better statistical and numerical modeling, and the availability of high-quality global datasets (primarily created through large international collaborations, which is a post-cold war thing). I also see a lot of room for improvement in our understanding of the coupling of mechanisms spanning vastly different timescales--for example earthquakes occur in seconds, post-earthquake phenomena last weeks to decades, the earthquake cycle lasts hundreds to thousands of years, and the cumulative deformation from earthquakes and related processes is what we call 'tectonics' over millions of year timescales. It's really hard to make a single numerical (i.e., finite element) model that works over all of these timescales and is driven by basic physics (i.e. an earthquake results from forces applied rather than being imposed). Nonetheless there are almost certainly a lot of really important coupling processes that occur on these different timescales but they are really hard to analyze (and a lot of interesting stuff happens at 20 km depth and at mm/yr rates, which is pretty damn hard to observe).

So I guess I see a lot of 21st century science as bridge building rather than outright discovery. This is fine. An analogy would be moving to a new country. The first bit is the discovery of the place, then you learn what language they speak. The learning doesn't stop there. As you learn the language, a lot of daily stuff makes sense. As you gain an understanding of the culture and the history, and can interact in a meaningful way, the value of that learning continues to increase, but it doesn't feel like 'discovery' like it did in the first year.

</comment><comment><USER42> Yet we directly observed gravitational waves, and compute power continues to grow.

</comment><comment> oh-kumudo Human have succeeded most of the applicable rules of our physical world. What left to be discovered requires much more time and effort.

</comment><comment><USER43> An unpopular possibly "devil's advocate" opinion: Considering that our scientific advances have led to us quickly using an enormous developed capacity to make the earth much less hospitable to human life, maybe a slowdown in scientific advances is not a bad thing.

On the other hand, yes, we are going to need some science to deal with what we have wrought without maximum misery. But I don't really trust us with it.

</comment><comment><USER44> What a joke. There are hundreds of thousands of papers published each year, and some very small faction of those (but certainly hundreds to thousands) make a bit impact on their field. But we summarize science based on three yearly prizes? Clearly science is not having enough impact on science policy writers.

</comment><comment><USER45> I find this article preposterous.

>While understandable, the evidence is that science has slowed enormously per dollar or hour spent.

The only evidence for this is Nobel Prizes won, split up by decade, and polled!? Biology, arguably one of the most exciting fields right now, doesn't even get considered in the Nobel Prizes! You _can_ get a Nobel Prize in medicine/physiology for biology-related efforts (see GFP), but there's no Nobel Prize for plant-related biology (that's what the Kyoto medal is for, which isn't mentioned here?!?)

I can't wait for science-illiterate politicians to take this ('Look at what YCombinator is saying!') and say we should defund science.

It has _never_ been as hard to get funding for your science as now - see e.g. [https://theconversation.com/with-federal-funding-for- science...](https://theconversation.com/with-federal-funding-for-science-on- the-decline-whats-the-role-of-a-profit-motive-in-research-93322) \- with huge issues (scientists living in precarious circumstances, ridiculously low salaries, attacks and defunding by conservative governments (US, Australia, Harper government in Canada) - writing an article _now_ saying that science is stagnant is foolish at best, dangerous at worst.

>Over the past century we've vastly increased the time and money invested in science, but in scientists' own judgement we're producing the most important breakthroughs at a near-constant rate. On a per-dollar or per-person basis, this suggests that science is becoming far less efficient.

A much simpler explanation is that we've picked the low-hanging fruits (think of how diverse the work of Darwin was! He picked up novel fossils literally on beach walks), now (especially in physics!) we need bigger and stronger efforts to go for the harder fruits.

<comment><USER46> >>Over the past century we've vastly increased the time and money invested in science, but in scientists' own judgement we're producing the most important breakthroughs at a near-constant rate. On a per-dollar or per-person basis, this suggests that science is becoming far less efficient.

>A much simpler explanation is that we've picked the low-hanging fruits (think of how diverse the work of Darwin was! He picked up novel fossils literally on beach walks), now (especially in physics!) we need bigger and stronger efforts to go for the harder fruits.

Your explanation that we have picked the low-hanging fruits is not mutually exclusive with the increased time and money invested in science. They would even be two sides of the same coin- as the easy pickings disappeared, we chose to invest more to get the harder fruits. As the trend continues, the search space for novel, useful discoveries becomes larger and larger, and the cost increases.

How do you prove to the politicians that a particular scientific investment is worth it?

<comment><USER45> >How do you prove to the politicians that a particular scientific investment is worth it?

I don't think that's possible. The history of science is full of science that shouldn't have worked but did - think of Barry Marshall drinking Helicobacter pylori, which no-one thought to be a causative agent, or think of Norman Borlaug, who came up with shuttle breeding (grow a plant twice a year by driving seeds around, at a time when people thought that you have to let the seeds rest for a while) - but the history of science is also full of things that should have worked but didn't, we just don't hear about those (classic survivor bias).

In some cases, you can tell whether an experiment is going to work and how it's important, especially if it's incremental work - in some cases, you simply cannot predict.

Just think of how delighted G. H. Hardy was that his mathematics was kind of useless to the general public, and how useful his work is nowadays (most importantly the Hardy-Weinberg equilibrium I guess?), he could not have predicted how that works out! What would he have written into his application for funding?

</comment></comment></comment><comment><USER13> I don't understand their methodology at all. Considering that the frequency of awarding nobel prizes remains the same , there is nothing that can be infered from their survey.

That said i agree with the premise of their article that science is not living up to expectations, and i believe this started somewhere in the 90s. For example consider the recent "Burden of disease" survey which was linked here a few days ago:

> GBD 2017 is disturbing. Not only do the amalgamated global figures show a > worrying slowdown in progress but the more granular data unearths exactly > how patchy progress has been. GBD 2017 is a reminder that, without vigilance > and constant effort, progress can easily be reversed.

[https://www.thelancet.com/journals/lancet/article/PIIS0140-6...](https://www.thelancet.com/journals/lancet/article/PIIS0140-6736\(18\)32858-7/fulltext)

This effect was probably many years in the making , and is only now becoming apparent. One possible explanation for the "sluggishness" in scientific output is that nowadays it is lacking new grand ambitious projects, in other words centralization and Big Science. The proliferation of PhDs has changed the way funding is allocated in recent decades for purely political reasons. It favors thousands of small grants that go to individual independent researchers who are all studying minute effects, are looking for results that barely pass the significance threshold and are publishing in order to build publication records to advance their career. The total number of researchers has doubled since the 80s , therefore this model may be actually detrimental to the process of scientific knowledge discovery at this time. For example, in my field of computational neuroscience, there is currently no equivalent of the "large hadron collider" project for the brain. Some years ago one such project, the 'Human brain project' was proposed and funded, which had a specific and ambitious goal: to simulate the entire brain. Academic politics however fundamentally altered the project and it is now a funding source for various kinds of ordinary research. To be clear, the project was probably ill- conceived from the start (imho), given that we don't have enough info to simulate the brain correctly, but regardless it was one of the few such efforts towards a singular ambitious goal. Subsequent funding schemes such as the US Brain initiative do not have such a focus. The ones who do undertake big science are private institutes like the Allen institute in their attempt to accurately map the entire brain. In any case i think public policy has to focus less on academia and more on science, in order to make progress otherwise we are bound to see detrimental effects a few years down the road.

</comment><comment><USER47> The graph should be adjusted by inflation, and probably by other factors, in order to accurately back the article's claim. $1M in 1940 would be equivalent to $17M in 2018.

<comment><USER48> The graph states that it is adjusted for inflation.

<comment><USER47> Ok thanks. I must have missed it.

</comment></comment></comment><comment><USER49> science having its own logistic curve ?

</comment><comment><USER50> A lot of the comments are listing their personal suspicion of causes, without attempting to illustrate or provide evidence. Some of those suspicions are probably true, and some false. Obviously it is also an outlet for all our personal gripes with the system. Irrespective of the veracity of a proposed cause, I would like to see more discussion of actual examples of current progress, and trying to identify what caused or enabled the progress to be made by comparing with the average paper in the back of our mind: what was different, why did they achieve the progress today and did nobody achieve the insight say five years ago?

Ideally, since we are discussing on HN, it should be an example that would be understood by most participants here.

It is in this spirit that I will give exactly such an example of recent progress (with which I am entirely unaffiliated).

First some minimal background which I assume you are _not_ familiar with: quantum chemistry and solid state physics software.

Just take a quick look at this list on wikipedia, you may recognize the names of some pieces of software like ABINIT... make sure you pay attention to the DFT column, and that virtually all packages support DFT calculations, which has been pretty much __state of the art for the last decades __.

[https://en.wikipedia.org/wiki/List_of_quantum_chemistry_and_...](https://en.wikipedia.org/wiki/List_of_quantum_chemistry_and_solid_state_physics_software)

In physics gradients often arise naturally for example forces as gradients of potential energy... But in general even outside physics gradients are useful in nearly all field for optimization...

Now the part you probably already understand: AD, Automatic Differentiation or Algorithmic Differentiation, and in the often occuring case of a single scalar function in N variables, _reverse-mode_ AD... And that it takes on the order of 5 times a single function evaluation to calculate a gradient.

Now specific subfields of physics have been using AD and adjoint sensitivities for a long time (nuclear engineering, oceanography) but it is not a standard part of physics curricula.

Outside of these specific subfields, __Automatic Differentiation has been gaining momenta over the last decades __(books, comprehensive reviews, ...)

Physics students of course learn differentiation symbolically on paper for short formulas, or in software packages like MACSYMA, Maple, ... but even then you keep the number of variables low for tractability. These students will also understand you can emulate differentiation numerically by using a finite delta:

d f(x1, x2, ..., xN) / dx2 = ~ [f(x1, x2 +delta, ..., xN) - f(x1, x2, ..., xN)]/delta

fully understanding that for a complete gradient you need (N+1) evaluations of f, that delta too large will be inaccurate due to functional nonlinearity, and delta too small will be inaccurate due to numerical rounding of floats... Nobody thinks of showing reverse mode automatic differentiation to physics students as part of their curricula! If you crash into a physics course and ask the students how to calculate the gradient of a big function of 1000 variables, they won't be able to help you, but then you can explain that such a thing is in fact possible!

You see where this is going...

What if some of the numerical computations, say molecular modeling, could benefit from this insight?

That's exactly what happened recently:

[https://arxiv.org/abs/1010.5560?context=cond- mat](https://arxiv.org/abs/1010.5560?context=cond-mat)

Now look at the authors' institutions of this paper:

SISSA, International School for Advanced Studies, 2 DEMOCRITOS National Simulation Center, Quantitative Strategies, Investment Banking Division, Credit Suisse Group

That's people with an interdisciplinary background.

Could it be that we are over-specialized? Or perhaps mis-specialized?

Imagine an alternate world where there is so much math that mathematics has decided to specialize thus: after a common course of Fundamental Math, you can specialize in Definitions, or perhaps in Theorems, or perhaps in Proofs, or perhaps in Conjectures... clearly their progress in math is going to suffer!!

Could it be that experts and professionals have become "too polite" to the point of circle j* ? A kind of "politeness omerta"? Imagine a mathematician and a physicist talking: as long as we're discussing math, the physicist nods with interest, and doesn't make suggestions how he would do it differently and vice versa. The "don't criticize a professional in his domain" attitude? Wouldn't it have been better if someone who understood AD say 20 years ago, who ends up in a conversation of molecular chemistry software physicist to say something like "I don't have the domain knowledge, but first thing I would do is find out if the calculation could be viewed as the computation of a gradient, or else as an optimization of an explicit goal" "how dare you make a suggestion outside of math?! and thats thousands of variables, its _intractable_ " ?

We might have had this 20 years ago! Mansplain it if necessary.

it's this upside down world where we are polite and stoic in our papers, but elbow working back stabbers undermining the workgroup next door, all the while feigning this "politeness professional omerta"</comment>


Privacy: Think Google knows your secrets? Meet Yodlee - <AUTHOR> http://money.cnn.com/2010/11/16/pf/online_banking_yodlee/index.htm

<comment><USER2> Yodlee's own free consumer webservice is outstanding.

<https://moneycenter.yodlee.com>

Once you enter your accounts, you see every transaction from yesterday aggregated in one place every morning. It's smart about categorizing expenditures, and presents you with a full portfolio. It predicts and notifies you of bills. It works with every financial account out there.

I've used the service for several years, with never an outage or a mistake. The UI steadily improves, version after version.

Most people use Yodlee's back end through one of its bank or brokerage partners, but using Yodlee directly and for free is definitely worth a try.

</comment><comment><USER3> Remedy: Always withdraw money from ATM and pay for everything with cash.

<comment><USER4> and try to always use the same ATM.

</comment></comment><comment><USER5> I think yodlee was started by one of the founders of hotmail.

</comment><comment><USER6> Never pay with debit

<comment><USER7> As opposed to credit? How would tracking that be different in this case?</comment></comment>


How the NSA scandal hurts the economy - <AUTHOR> http://globalpublicsquare.blogs.cnn.com/2013/07/15/how-the-nsa-scandal-hurts-the-economy/

<comment><USER2> For what it's worth - I've got clients and colleagues asking hard questions about data jurisdiction and data sovereignty in the last 6 weeks. The old "obvious answers" to cloud storage and computing are no longer considered quite so straightforward. Can an Australian company be considered to be doing "due diligence" if storing customer data on Amazon's cloud? Or Azure? Or AppEngine/Heroku/Rackspace/Force.com? I've even got clients asking if there are MessageLabs equivalents that're solely under Australian law instead of US law?

Personally, I don't trust the Australian Security Intelligence Organisation to be any less morally corrupt that the NSA - though I hold some small comfort from the fact that they're significantly less well funded and almost certainly don't have the small country sized staff of Math PhDs that the NSA have…

Even though I don't assume we (here in Australia) have any better protection against government snooping - everyone I talk to seems to understand that acknowledging and accepting the risk of local government snooping is something easily explained to customers/shareholders/judges than having to explain why you stored confidential information on cloud services that you knew were under foreign surveillance.

My response has been to (bring forward the implementation of) "encrypt all the things". I suspect this is already costing US businesses - I wonder if Dropbox/GDrive/SkyDrive are noticing reduced de-dupe rates? I'm now using encfs for most of the free storage they're giving me.

</comment><comment><USER3> I tried this argument (hurting USA's economy) as an argument to several friends on how harmfull the NSA sweep everything up surveillance will probably hurt our economy. They don't buy my argument because the US news media tells one side of the story only (mostly). It only takes a few seconds to find relevant foreign news stories, but there seems to be a hysterical blindness happening.

I enjoy playing Chess and Go, and it is very important to accept what is on the board as reality, and play the game accordingly.

I listened to Richard Stallman earlier today, saying that now is the time when perhaps the fight for freedom could work. I think Richard is perhaps overly optimistic, but we can hope. I have increased donations to EFF, FSF, and the ACLU - that seems the best I can do. I used to write frequently to my elected representatives but most of them seem to have their own agendas that are not aligned with what I think are public interests.

</comment><comment><USER4> I've said this before, but I'll repeat it here. There is zero chance that anybody will believe what the US government / US companies will say from now on. Their credibility has been fatally damaged, for years to come.

The fact that we are not yet seeing any exodus from the current cloud offerings is because there are no easy-to-use alternatives which, _at the same time_ , offer a better degree of trust. But I am confident they will emerge, since a huge market for these kind of services has opened up - before the Snowden relevations we were just suspicious, now we are certain.

And an important selling point of these services will be to not be US-based, and better yet, not run by any american company. Trust can not be regained, and _whatever_ will be said won't be believed: so you say that you are not spying on me? Yeah, sure! So you say you have no backdoors? Yeah, sure! So you say you can guarantee under oath that you will not spy on me, under any circumstances, and you can prove this with court documents, contracts, laws passed by the US congress, and the clear and concise statements of the US president and the CIA boss? Yeah, sure!

</comment><comment><USER5> It's sad that this is our best chance for changing US policy at this point, but I'll take it. I would have hoped that the ideals of freedom were something that the American people embraced for their own sake, but you play the hand you're dealt.</comment>


A VC Gone Mad? - <AUTHOR> http://web1979.wordpress.com/2007/07/27/a-vc-gone-mad/ So it's time to revisit Twitter.<p>The news today that Fred Wilson's Union Square Ventures has funded Twitter is enough to bait me back into the ring. <comment><USER2> Fred Wilson is basically following the tried and true investment philosophy of Peter Lynch - invest in what you know.

He uses twitter. Lots. Why wouldn't he invest?

</comment><comment><USER3> A stab at twitter from the man who days earlier recommended buying facebook. What's the fb killer business model again? rigghht...

</comment><comment><USER4> this blogger lost me when he implied the delicious acquisition was a mistake. that was a brilliant acquisition, and _cheap_. like in the 20M - 30M range for a very important internet property. it would sell for a lot more than 30M now.

</comment><comment><USER5> Their business model, as far as I can tell, is to license the presence technology/platform (to other web2.0 sites, mobile carriers/developers, etc), and become the "finger" of web2.0. They want to become a network backbone, after a fashion.

</comment><comment><USER6> I thought the "blame the buyer" comment was an interesting take on the relevance of business models.

It's essentially the crazy-uncle-locked-in-the-attic aspect of all this, which no one wants to talk about.

</comment><comment><USER7> Are social networks becoming the next the dot.com bomb?</comment>


Tesla becomes most valuable automaker in latest stock rally - <AUTHOR> https://www.reuters.com/article/us-tesla-shares/tesla-becomes-most-valuable-automaker-in-latest-stock-rally-idUSKBN2426E8 <comment><USER2> I don't understand the mentality people have when calling Tesla an automaker. Automakers today traditionally outsource a majority of their parts and are just glorified assemblers. Meanwhile tesla has much more vertical integration in their cars and owns much more of the stack, from the batteries (there are even rumors they'll go into mining to secure steady supply), to their software and self driving stack, to grid storage, etc.

I'm not sure exactly what I'm getting at here, but it reminds me of back in the day when people called amazon just a bookstore.

<comment><USER3> I think you're twisting reality a tiny bit. Car brands have used all kinds of partnerships from car body design, to engines. Just last week I learned that DMC Delorean use a PRV engine. PRV standing for Peugeot Renault Volvo. 3 European brands, nothing American in it..

Same goes for airplanes, boeing and airbus uses parts from similar suppliers.

<comment><USER4> I don't see how this is a criticism given that Tesla is actually vertically integrated.

<comment><USER3> integration probably varies over time, is it fair to take it into account ?

</comment></comment></comment></comment><comment><USER5> Calling the market is a fool's game, but that valuation makes no sense to me at all.

Remember this is supposed to be the expected value weighted across up and down scenarios. The down scenarios are clear, so in what crazy world are the up scenarios enough to average out to 210 billion?

<comment><USER6> The up scenario is basically the one where Tesla's cars are Apple's Macs. Apple used the success of the mac to get itself into a number of other industries to such an extent that people now fear it will abandon the mac because it makes such a small amount from it.

So what are Tesla's other business lines? Solar panel deployments (residential and industrial), battery deployments, electrical utility software providers, autonomous robotaxi services, residential HVAC. This is just things they've publicly said they're working on today. It would surprise me if they don't have an electric plane in the works by the end of this decade. Maybe industrial sabatier-process based carbon capture for jurisdictions with carbon taxes and Mars bases.

Basically, they have enough "bets" that you can believe, if you want to.

<comment><USER7> The comparison to Apple makes sense: I have spent the last decade and a half reading prognosticators announce that Apple's stock is overpriced, and so far all of them have been wildly wrong.

Eventually they'll almost certainly be right, since, historically speaking, no stock has gone up forever, but they've been impressively wrong so far.

</comment></comment></comment><comment><USER8> Generally market bubbles come to an end when you run out of fools, but in this case the supply of fools seems just limitless. What's the story here? Where are they coming from? Is it massive leverage? Is it free money from the fed? What accounts for the endless supply of idiots? I really do want to know who is buying at these prices and where they are getting the cash to do so.

</comment><comment><USER9> The valuation makes sense. In the coming years the auto market is going to be moving rapidly to ev's and away from ice's. Demand for ev's is going to outrun supply, in part due to battery shortages.

Tesla is expanding rapidly and is going to be in a better position than any other car maker to take advantage of the situation, though lately VW has also getting very serious about ev's.

</comment><comment><USER10> Nikola is now bigger than Ford, Fiat/Chrysler, Hyundai, Suzuki, Nissan, and Subaru

<comment><USER11> Thats absolutely nuts, considering they yet to sell any cars, or book ANY revenue.

People bashed Tesla for years because they weren't profitable... But at least they had revenue.

<comment><USER12> At today's NKLA valuation, Tesla was selling 20K cars a year. And Tesla had already proven the BEV are far better than FCV in every aspect (well explained by VW: [https://www.volkswagen-newsroom.com/en/stories/battery-or- fu...](https://www.volkswagen-newsroom.com/en/stories/battery-or-fuel-cell- that-is-the-question-5868))

</comment></comment></comment><comment><USER3> who's buying which rumor ? battery day ?

</comment><comment><USER13> Impressive given that it has never made a profit!

PS: [https://www.theguardian.com/technology/2020/jul/01/tesla- bec...](https://www.theguardian.com/technology/2020/jul/01/tesla-becomes- worlds-most-valuable-carmaker-without-making-a-profit)

<comment><USER14> Is it? This was true if Amazon for well over a decade, and look at them now.

If you can meaningfully reinvest into your business rather than show a profit (and pay taxes on it), then you're in a good position. Tesla has the growth that merits doing this, which is why their share price is so high.

<comment><USER15> Tesla's operating margin (16 years after operation) 0.3%, Amazon's operating margin in 2010 (roughly after 16 years of operation) 4.4%</comment></comment></comment>


The Open Problem Garden, a collection of unsolved problems in mathematics - <AUTHOR> http://www.openproblemgarden.org <comment><USER2> Why doesn't this project utilize some variety of TeX typesetting to make the notation a bit more legible? Drupal has a number of modules that will do the trick.

[https://www.drupal.org/node/159031](https://www.drupal.org/node/159031)

<comment><USER3> It looks very much like there was or is a Tex/LaTex module installed but it's not working right. I'm seeing LaTex style markup all over the place...

</comment></comment><comment><USER4> I've emailed them to suggest MathJax, and to ask if they're intending to be running something. If they are, it's broken. If they're not, MathJax is superb.

No idea if they'll respond, but emailing them seemed more useful and productive than just complaining about it here.

</comment><comment><USER5> I'm not much of a TeX interpreter, so all that ${\ stuff is extremely irritating. I find it unreadable.</comment>


VW, Daimler and BMW conspired to limit dev and rollout of emission cleaning tech - <AUTHOR> https://www.dw.com/cda/en/european-commission-finds-german-automakers-illegally-colluded-on-emissions-technology/a-48218578 <comment><AUTHOR> Main findings of the investigation by EU's antitrust regulator (article extract):

* From 2006 to 2014, the three German automakers conspired to limit the development and roll-out of emission cleaning technology for passenger cars sold in Europe.

* The talks were aimed at restricting competition and "breached EU antitrust rules."

* Two types of technology specifically were restricted: one to reduce nitrogen oxide emissions from diesel cars, and another to reduce harmful particulate matter from petrol engine cars.

* The companies "denied consumers the opportunity to buy less polluting cars" despite the technology being available.

IMO: the Dieselgate is still known as the "Volkswagen emissions scandal" (e.g see the title of the main Wikipedia article on the subject) and yet, the deeper we go down the rabbit hole, the more this appears to be an industry- wide conspiracy. Is there any other case as global and catastrophic as this one, in terms of collusion against regulations, people's health and the environment?

</comment><comment><AUTHOR> >A Daimler spokesperson said that they were cooperating with investigators, but that the company does not expect to be fined as a result of the probe.

Why?</comment>


Toward a Predictive Theory of Depression - <AUTHOR> http://slatestarcodex.com/2017/09/12/toward-a-predictive-theory-of-depression/ <comment><USER4> Professor of psychiatry here - I've spent the past 20 years trying to help folks with depression, and related illnesses, and trying to study these diseases at a genetic level. I suspect this will get me down-voted as I know the medical/biological perspective is often unwelcome on hn. However: a key error people make here and elsewhere in speculating about depression (and suicide, and ADHD, and...) is either extrapolating only from their own experience, or from a vastly oversimplified psychology 101 perspective. For example: "Depressed people move more slowly, in a characteristic pattern called "psychomotor retardation". Well, yes - except when they have agitated depression, characterized by psychomotor restlessness. Or, when they have neither of these. Likewise, oversimplifying by claiming it is a disease of dopamine, or serotonin, doesn't much help. When my car stops working, is it a disease of gasoline? Part of the challenge of depression is its very heterogeneity. What we need is not more theorizing, it is more recognition that we need better treatments of all sorts.

<comment><USER5> I've never been sure why it's treated as a disease rather than a symptom. It's like saying someone with a high temperature is suffering from fever. Which is nice and descriptive and treatable with the right medications, but does nothing to describe the underlying cause.

<comment><USER6> Lots of what we now consider symptoms were thought of as diseases until the underlying problem was better understood. It wasn't that long ago that your fever example was how people thought. The fever was the disease regardless of the unknown underlying cause.

That depression is considered a disease by itself and not a symptom of many, possibly related, problems suggests how little we understand it.

</comment></comment></comment><comment><USER7> It's a weird coincidence to read this. I was doing basic research on depression and saw that dopamine played a role in helping people feel good. One WikiHow article [0] later, I had an idea: Start reaching goals. I started very small, like taking progressively longer walks.

Since that time I have earned something like five different certificates in various areas of interest (ARRL coming up next) and as a very recent development, completed my university degree after 15 years of delay due to, among other things, depression.

I find that depression can still affect me, but the episodes don't last as long (around 2h max, lately) and I have a bunch of tools for dealing with them.

[0] [http://m.wikihow.com/Increase-Dopamine](http://m.wikihow.com/Increase- Dopamine)

<comment><USER8> This works very well for low-grade depression for sure. There seems to be a threshold where this type of intervention becomes completely ineffective (personal experience).

<comment><USER7> I agree. From keeping logs over the last few years, severe depression was more likely to occur due to e.g. not enough sleep or just illness, and in those cases accomplishments were impossible. Rest was it, and I felt lucky if I could get good rest before acting on any depressing thoughts or ruminations. So I wouldn't call goal-reaching a standalone cure; far from it. Also there's the fact that people accept/reject potentially helpful information based on different criteria, from very rational (is it in a reputable journal) to irrational (do I like the person who gave the information) to incredibly subjective (did I think of it on my own?). So the idea of "why not try accomplishment" may be doomed from the start, due to factors outside of the advisor's control.

<comment><USER8> I noticed that rest only somewhat helped, but was significantly more effective if absent of any worry of external responsibilities. E.g. If I took vacation from work and didn't tell my girlfriend about it so she wouldn't worry about me and text me incessantly... I could turn off the thoughts/worries of the outside world completely for a few days and that made a pretty dramatic difference. I didn't cure me at all, but it took me out of the really dangerous zone after about 2 days. Also, sleeping in a hammock, haha. So much better than breathing stale air for 2 days.

This is more circumstantial however, as my trigger at the time was accumulating too many perceived responsibilities and fearing failure. To be honest I feel the triggers are nearly irrelevant, but removing myself from them gives me enough temporary relief for my body to get out of stress mode. Which in turn removes some of the depressive overhead to give myself a golden window of opportunity.

</comment></comment></comment></comment><comment><USER9> > Depressed people move more slowly, in a characteristic pattern called > "psychomotor retardation". They display perceptual abnormalities. They're > more likely to get sick. There are lots of results like this.

> Depression has to be about something more than just beliefs; it has to be > something fundamental to the nervous system.

Depression is frequently related to metabolic problems: people are exhausted because their mitochondria aren't putting out enough ATP to run their nervous system.

There are various interventions that can help. B-vitamins are important cofactors for burning sugar to make energy. T4 thyroid is activated by the liver to make T3, which is one factor that boosts the metabolism. If the liver is overworked (alcohol, drugs, hormonal imbalance, too much pufa, etc), it can have trouble activating adequate thyroid.

Emotional resiliency is a big factor here too. My friend was really just lonely before she met me. She's doing better now because she now has a 'vision' for her future (she originally 'lost her future' when she was expelled from high school, essentially for being depressed and self-medicating with drugs that are now being investigated for treating depression).

<comment><USER10> there's a difference between doing a bunch of molly every weekend for fun and self medicating.

<comment><USER11> There is, it's usually pretty easy to tell the difference. Doubt anyone would mistake them, and I doubt the OP is.

<comment><USER9> > There is, it's usually pretty easy to tell the difference. Doubt anyone > would mistake them, and I doubt the OP is.

A few months ago a young man (early 20's) came into a shop I sometimes hang out at. "What brought you to town?" "Recovery." He'd been in town for 3 weeks, and sober for 3 weeks and 1 day.

I asked him, "People usually know when their problems when substances started, when did it start for you?"

When he was in 3rd grade his teacher thought he was disruptive. They put him on some kind of medication for "ADHD". He told his parents in 9th grade that he was going to kill himself if they didn't take him off the pills. Soon after discontinuing his "ADHD" medications was when he started experimenting with self-medicating... Tobacco, alcohol, etc.

His latest bender was on alcohol & laughing gas (nitrous oxide [0], for the euphoria).

[0] [https://en.wikipedia.org/wiki/Nitrous_oxide#Recreational_use](https://en.wikipedia.org/wiki/Nitrous_oxide#Recreational_use)

After a bit I told him, "I bet you're exhausted."

"YEAH!"

"Recovery" is good for getting people to feel like failures, so I thought it'd be helpful for putting a different 'frame' on his last 15 years. I told him that he doesn't actually have a substance abuse problem, he's just been exhausted this whole time, and didn't know what to do to help improve his energy levels. I gave a few suggestions for keeping his energy levels up.

Saw him three or four more times over the next 2-3 months, and he was doing much better. I saw him last 2 days before he returned to where he came from. I think he's got a chance this time.

<comment><USER12> Would you mind to elaborate a little bit on your suggestions? That is if it's not too specific.

<comment><USER9> The only thing that fellow could really do for himself was buy some coconut oil. The brains of alcoholics become insulin-resistant and switch to running on acetate, one of the breakdown products of ethanol. The medium-chain- triglycerides in coconut oil provide brain cells with an alternate source of fuel. (I understand that Coconut fatty acids are short enough to be burned without having to go through another step that longer saturated fats [butter, etc] require; I don't remember exactly what that step is.)

I don't think that guy was able to get the coconut oil while at the recovery center. I think giving a more helpful context for viewing his history of addiction was much more helpful than the recommendation for coconut oil.

Another heavy drinker I know does fine while on "vacation", but "stress" in his every day life leads to excess drinking. I have no influence with this one, but I'd try to get him to eat more sweets when in stressful situations - juices/etc. I've heard an old treatment for alcohol withdrawal is to feed them lots of sugar. Here's a random NY Times article that says sugar helps reduce cortisol levels: [https://well.blogs.nytimes.com/2015/04/23/sugar-as-a- stress-...](https://well.blogs.nytimes.com/2015/04/23/sugar-as-a-stress- reliever/)

IMHO, sugar in juices and other foods is basically fine; starch is 100% glucose and is a greater contributor to excess weight. Lots of caveats with regards to the sugar issue...

<comment><USER13> I was a heroin addict for 6 years, and have been clean for 5. The sweets thing was very true for most addicts I knew as well as myself: in withdrawals I'd crave sugar something fierce. Gummy bears would temporarily make me feel better. It was interesting.

</comment></comment></comment></comment></comment></comment></comment><comment><USER14> When I was depressed I had perfect trust in my ability to predict events, it was just that no matter how much I tried I couldn't predict a future where I weren't constantly in agony. I got over it by letting go of most things that were hurting me like friends, family, dreams etc. So now I live a fairly unassuming life but I am content. It might sound a bit boring but it is much better than living in agony every day for the rest of your life.

<comment><USER15> How did you let go? I've tried things like meditation, but it always feels like things keep coming back.

<comment><USER14> It wasn't something I just decided to do, 15 years of hating my life eroded almost everything but my basic needs. Now I can look back at that time and with confidence say 'no' when asked to do something which would cause me to relapse.

</comment></comment></comment><comment><USER16> What I have found through my experiences with depressed people is that the vast majority of them are not insane people in a sane society, but perfectly sane people in an insane society, people who have maladapted to their environment but would otherwise (if they had a better one) be perfectly healthy individuals. Many of them internalize their sadness and helplessness and make it a part of their identity, further reinforcing the common belief among depressed people that they are ill at a biological level (and are not merely healthy people in an unhealthy environment). I don't think the solution to this is ever going to lay in the hands of psychologists, and certainly not psychiatrists (SSRI's and SNRI's work only slightly better than chance in most people, and taking other antidepressant/mood-stabilizing medication is a hellish experience all its own) - it lays in the hands of the people who construct the environment we live in, the people in charge of its development and health, the people who have the most power over society. These are the people who have destroyed it and made the environment we all have to live in such a hellish place, and their successors will certainly be the only people who can fix it.

<comment><USER17> I don't know. Depression has been recorded for thousands of years, so either society has always been insane (quite likely), in which case it's not likely to turn around any time soon, or there are other causes of depression. Some people can also fall into a depression for no identifiable reason (I can vouch for that from experience).

SSRIs and SNRIs and other medications have saved my life and some of my sanity, and I have high confidence that it's not a placebo - for me - <USER3> tried and failed with probably a dozen of these over the years, before hitting on a workable combination.

And after maybe 5 years or so, that combination became ineffective, and I had to try a few more.

But that's just me, so this is anecdata.

<comment><USER16> >I don't know. Depression has been recorded for thousands of years, so either society has always been insane (quite likely), in which case it's not likely to turn around any time soon, or there are other causes of depression. Some people can also fall into a depression for no identifiable reason (I can vouch for that from experience).

Of course there have always been depressed people, but the modern phenomenon where supposedly 10% of the population will have it in their lives, or where entire countries experience skyrocketing suicide rates, is certainly not related - I find it very difficult to believe that they all have a biological problem, instead of a cultural or societal problem.

>SSRIs and SNRIs and other medications have saved my life and some of my sanity, and I have high confidence that it's not a placebo - for me - <USER3> tried and failed with probably a dozen of these over the years, before hitting on a workable combination.

I don't doubt this - I only said that the vast majority of sufferers of depression do not seem to suffer from a biological problem. From my personal experience, I have seen that most of the depressed people I have known have gotten better not through medicine or medical procedure, but by consciously trying to make their lives better and be the best person they can be. Obviously this won't work for everyone - there really is such a thing as biologically-caused mental illness (see: schizophrenia, for example), but I think it is much rarer than people are willing to admit.

>But that's just me, so this is anecdata.

All data is anecdata. It's not worthless just because it happens to be your actual life and not collated and tallied by some detached scientist, it has meaning and value even without being a part of some anonymous dataset you can take confidence intervals on.

<comment><USER17> You make your points persuasively and elegantly, and I agree with you for the most part. I think much of what is diagnosed as depression in recent times may be a synonym for despair at these times (echoing Cicero : "O tempora! O mores!"). On the other hand, clinical depression of the kind that has assailed me since my teenage years (and from which my son suffers and showed signs of since 5 or 6 years old) appears most certainly to be genetically caused.

I suspect that maybe a physical cause of depression is less rare than you think.

</comment></comment></comment></comment><comment><USER18> > Depression has to be about something more than just beliefs; it has to be > something fundamental to the nervous system.

Is he saying thoughts and beliefs aren't fundamental to the nervous system/CNS/brain?

What if people get depressed because their lives suck and they had a shitty childhood?

Also, why are women twice as likely to get depressed than men?[0] If "happiness is the derivative of neural confidence"... then do men have more neural confidence?

[0] [http://www.mayoclinic.org/diseases- conditions/depression/in-...](http://www.mayoclinic.org/diseases- conditions/depression/in-depth/depression/art-20047725)

<comment><USER19> > What if people get depressed because their lives suck and they had a shitty > childhood?

But people don't only get depressed because of those things; plenty of people's "lives suck" who aren't depressed, and plenty of people with good lives _are_ depressed.

[https://en.wikipedia.org/wiki/Major_depressive_disorder](https://en.wikipedia.org/wiki/Major_depressive_disorder) says:

"The cause is believed to be a combination of genetic, environmental, and psychological factors.... About 40% of the risk appears to be related to genetics."

So yes, thoughts and beliefs are part of depression, but they certainly don't explain all of it.

<comment><USER20> Beliefs are passed down like genes.

<comment><USER21> This is true in the case of my family. My brother and I learned how the world worked from my parents, and it's taken years for me to unlearn what they taught me instead of some actually useful beliefs.

</comment></comment></comment></comment><comment><USER22> This doesn't make sense to me. Pretty much every purported treatment for depression involves either increasing the amount of sensory stimulation you're getting or else becoming more attuned to sensory stimulation: exercise, acupuncture, aromatherapy, drugs, meditation, float tanks, social interaction, music, intensely flavored foods, cold therapy, saunas, electroshock therapy, etc.

Occam's razor would suggest its more likely that lack of sensory stimulation causes some sort of brain damage, e.g. inflammataion or receptor density changes, and that it just takes a few weeks to reverse. I'm not seeing what evidence there is for this pereptual confidence model beyond what I'm proposing. I wouldn't doubt if people do get more confident in some aspects of their sensory experience, but it would seem difficult to prove this as an explanatory model.

<comment><USER14> > exercise, acupuncture, aromatherapy, drugs, meditation, float tanks, social > interaction, music, intensely flavored foods, cold therapy, saunas, > electroshock therapy, etc.

None of those are guaranteed to work though, a significant number of depressed people never gets over it and it wouldn't be that way if we knew effective treatment methods.

<comment><USER23> The current answer is we don't yet have full functional pathologies of types of depression, noninvasive, specific mood disorders classified down to exact neurological dysfunctions, accurate and precise knowledge of how existing antidepressants work and evidenced-based application of particular therapies based on such scientifically-identified presentations. Clinical psychiatry has been mostly a scattergun, guesswork that lacks scientific rigor.

For example, I just tapered-off mirtazepene as it stopped working after 9 years and, so far, bupropion isn't helping. I'd really like to try S32212 if it proves safe enough in human models because it likely lacks the GI and weight-gain issues.

<comment><USER24> I've tried almost every anti depressant on the market and none of them worked. What did work for me is kratom. I've been taking it daily for about six months and it has completely turned around my mood and general enthusiasm. There are some minor side effects but they went away after six weeks or so. I know it might frowned upon to recommend what is basically an opiate for depression, but it has been a minor miracle for me.

<comment><USER20> And what if your depression is cause of beliefs/behaviors and not curable with pills?

</comment></comment></comment></comment></comment><comment><USER23> Umm, not sure this is universal but partial. Depression seems evolutionary psych for multiple, overlapping purposes:

\- Reducing aggression (and hence conflict) in larger groups.

\- Focusing on existential re/purpose by turning inward (ie find a new job/hobby/outlook/philosophy). Lots of reading, writing and ideation. Hopefully rebuild confidence/positivity with completing something.

\- Retreating for terminal resource consumption minimization. Leave more for the young.

Also, anecdotally, I did everything "right" (ie CBT, frequent vigorous exercise, solid career, friends, sleep hygiene, address sleep apnea) and still couldn't escape depression's "mind pain" and depressive state. Only medication worked, like parting the clouds, waking up refreshed and organized thinking instead of a jumble.

<comment><USER20> Medication will stop working, it always does.

<comment><USER25> That's the spirit!

In all seriousness, what else is there to do? Take whatever relief you can get while you can get it.

<comment><USER20> Remove bad habits/ behaviors/people... cure the problem

<comment><USER17> So you are hoping there will be an ELE for the human species? Because that's the only way your prescription is going to cure the problem on a global basis for the foreseeable future.

<comment><USER20> When I say remove I didn't mean remove from existence lol, just from your life. It's true this isn't a solution that can be implemented by governments, it has to be implemented by the patient.

</comment></comment></comment></comment></comment></comment><comment><USER26> >One complicating factor – how do we explain depressed people's frequent certainty that they'll fail? A proper Bayesian, barred from having confident beliefs about anything, will be maximally uncertain about whether she'll fail or succeed – but some depressed people have really strong opinions on this issue. I'm not really sure about this, and admit it's a point against this theory. I can only appeal to the math class example again – if there was a math class where I just had no confidence about anything I thought or said, I would probably be pretty sure I'd fail there too.

I think the model actually fits here better than the author realizes. Predicting failure is much safer than predicting success, it's erring on the side of caution. If you predict that you will fail at something and prepare for the consequences of that failure, but succeed instead, you are better off than if you had predicted success, prepared for success, and failed. To me, it makes sense that, when uncertain, the default would be to predict failure.

And later:

>One problem with this theory is the time course. Sure, if you're eternally successful, you should raise your confidence. But eternally successful people are rarely eternally happy. If we're thinking of happiness-as-felt-emotion,itt seems more like they're happy for a few hours after they win an award or make their first million or whatever, then go back down to baseline. I'm not sure it makes sense to start lots of new projects in the hour after you win an award.

Wouldn't a rise in confidence have a corresponding rise in expectations? New goals mean a return to the state of "haven't succeeded yet".

Personally I'm finding this model seems to fit better than any other model I've heard of. Of course, I'm just some idiot on the internet, not a psychiatrist.

<comment><USER24> This sounds a lot like Martin Seligman's work. [https://en.m.wikipedia.org/wiki/Martin_Seligman](https://en.m.wikipedia.org/wiki/Martin_Seligman)

</comment></comment><comment><USER27> It's great to see this here.

For context, this is the model the author is referencing:

[https://slatestarcodex.com/2017/09/05/book-review-surfing- un...](https://slatestarcodex.com/2017/09/05/book-review-surfing-uncertainty/)

Scott Alexander is currently my favorite essayist. I just wasted most of yesterday reading the last 18 months of his "links" posts.

I regularly pass his work on to friends and would do so more often if I didn't think it would upset people.

</comment><comment> david-cako This particular topic is very much of interest to me; I've spent a good amount of time thinking about how depression relates to ADHD.

Depression and ADHD are both related to dopamine and norepinephrine dysfunction, but depression has a strong element of serotonin dysfunction as well.

One of the defining characteristics of hyperactive individuals is their addiction to stimulation (work, learning, leisure), which to me seems like it's fed by negative reinforcement. Removal of these things sends them into a depressive state because there simply isn't anything keeping them excited (and perhaps distracted from neuroticism).

It seems like depression is kind of the contrapositive. There is the same element of dopamine dysfunction, spawning disinterest and hopelessness, but they lack the positive reinforcement of _anything_ to pull them out of. I think this is where the sprinkling of serotonin dysfunction into the pie comes in. There's a very low level of security or confidence in any sort of endeavor.

</comment><comment><USER28> I typically have odd thoughts that are often wrong. But I really like quote from 'The Martian', "let's science the shit out of it".

I don't know if it's possible to science your way out of depression. If in some way the curiosity gives a small enough lift to keep you getting one foot in front of the other and learning about what's causing your depression.

It at least seems relatively risk free way of tackling depression. Learn about the beast and track what you do and what changes your mood.

But is this something you can tell people that are seriously depressed or is it just another useless bit of advice?

<comment><USER17> > But is this something you can tell people that are seriously depressed or is > it just another useless bit of advice?

From my point of view, it's not useful advice, partly because severe depression utterly saps the motivation to do virtually _anything_ , and partly because depression is highly resistant to rationality.

Mandatory references to "Hyperbole and a Half"[1][2].

[1]: [http://hyperboleandahalf.blogspot.com/2011/10/adventures- in-...](http://hyperboleandahalf.blogspot.com/2011/10/adventures-in- depression.html)

[2]: [http://hyperboleandahalf.blogspot.com/2013/05/depression- par...](http://hyperboleandahalf.blogspot.com/2013/05/depression-part- two.html)

<comment><USER28> Yes, this wouldn't work for the most serious cases of depression. It might help for those who are at a stage where they're willing to try something.

</comment></comment></comment><comment><USER29> I'm not sure why none of the studies touch on seasonal variation. I think I will research more of that but does anyone has a reference?

<comment><USER30> This? [https://en.wikipedia.org/wiki/Seasonal_affective_disorder](https://en.wikipedia.org/wiki/Seasonal_affective_disorder)

</comment></comment><comment><USER31> No mention of sleep?

* [http://www.thelancet.com/journals/lanpsy/article/PIIS2215-03...](http://www.thelancet.com/journals/lanpsy/article/PIIS2215-0366\(17\)30328-0/fulltext)

<comment><USER32> I think there's a lot of agreement that sleep is important, but not much consensus around models that are useful for treating it. Right now there's a big push for CBT-I, but the evidence is mostly low-quality and not really focused on anything except reducing subjective insomnia severity (this study is better than most in that respect).

</comment></comment><comment> ainar-g A book review[1] linked in the article is also very interesting.

[1] [http://slatestarcodex.com/2017/09/05/book-review-surfing- unc...](http://slatestarcodex.com/2017/09/05/book-review-surfing-uncertainty/)

</comment><comment><USER33> Chekroud (2015) here - just wanted to say that I think SSC is awesome, and does a terrific job discussing complex problems in psychiatry & philosophy.

<comment> ainar-g Could you give any recommendations of great articles on that blog? Or should people just go to the About/Top Posts[1] page and go down the rabbit hole from there?

[1] [http://slatestarcodex.com/about/](http://slatestarcodex.com/about/)

<comment><USER34> Here's a reddit survey of people's favorite posts, which closely matches my own:

[https://www.reddit.com/r/slatestarcodex/comments/61qrm2/favo...](https://www.reddit.com/r/slatestarcodex/comments/61qrm2/favorite_post_on_ssc_survey_2017_results/)

If your main interest is psychology, though, note that many of the top posts are more in the sociology and politics realm.</comment></comment></comment>


Natural language parsing for the web - <AUTHOR> http://nlp.naturalparsing.com/browserparser/parse

<comment><USER4> So I was out to dinner tonight and got an email from my VPS saying my disk IO was high. At first I thought someone might have hacked my server, but it turns out that someone posted the link to my site on HN!

I've been working on this website for the past couple of weeks. Please email me at andrew [AT] naturalparsing [DOT] com if you have any questions or suggestions, or want to use the API.

FYI, right now the API is not using all the capabilities of the Stanford Parser, just the word tagging part. More features will be implemented soon. Let me know if you have any specific requests.

Andrew

<comment><USER5> Have you seen: <http://www-tsujii.is.s.u-tokyo.ac.jp/~tsuruoka/postagger/>

It's a POS tagger whose output is extremely close (read: identical) to the Stanford POS tagger.

It's also much faster. I was able to tag a corpus of 300K sentences with this in 15 minutes. With the Stanford POS tagger it took the entire weekend.

Sadly, this tool's license does not allow commercial use and it is not released under the GPL license.

</comment></comment><comment><USER6> Parser spins and spins with no output for me. Have we already overloaded it?

</comment><comment><USER7> I'm waiting for the day when fruit flies finally get recognition as a species and fruit stops flying ;-)

<comment><USER8> I cannot recommend this comment highly enough! I urge you to waste no time to read this comment over and over again.

</comment></comment><comment><USER9> I seem like it's just a part of speech tagger, rather than a parser.

<comment><USER10> Wouldn't that make it a lexer then?

<comment><USER9> I suppose so, but for human languages it's much more difficult, since it assigning a type to a token is context dependent.

<comment><USER11> And there are unknown words (words that are not covered by the lexicon).

But the top-poster is right: the linked website does part of speech tagging, not parsing.

Providing a wide-coverage parser for the web is still hard. The number of possible parses for long sentences is enormous. Even if a sentence is not ambiguous to us, grammars allow all kinds of ambiguities.

There is a web demo for a Dutch system that is developed in our research group[1], but heap size and time limits are used, to exit gracefully if parsing takes too much time/memory (and sentences with more than 20 words are ignored, since you'll really want to do offline parsing).

[1] <http://www.let.rug.nl/~vannoord/bin/alpino>

<comment><USER12> Statistical methods do quite a good job of pushing that ambiguity back under the rug (see link to Berkeley parser further up, or the work on statistical disambiguation in Alpino).

As to the time consumption and complexity, that's a known problem of unification grammars (or just any grammar that does a little bit more) - <USER2> see this paper by Matsuzaki et al for efficient techniques to speed this up: [http://www-tsujii.is.s.u-tokyo.ac.jp/~matuzaki/paper/ijcai20...](http://www- tsujii.is.s.u-tokyo.ac.jp/~matuzaki/paper/ijcai2007.pdf)

<comment><USER11> True, but you still have to build up a forest from which every parse can be extracted (as Alpino does). Of course, it does reduce the cost of ambiguity. E.g., I implemented packing in the Alpino chart generator, and there are very many edges with the same 'semantics' that can be packed, especially since Alpino allows for a lot of optional punctuation, and there are often roots plus subcategorization frames that give multiple inflections. With a beam search, unpacking is a lot more efficient.

There are also many other possible optimizations. E.g. with a left-corner parsers you can exclude left-corner spines that will probably not lead to a probable parse (Van Noord, Learning Efficient Parsing, 2009, and other work). And, of course, reduction of lexical ambiguity by restricting frames using a part of speech tagger.

Still, even in this best-1, optimized scenario, _real-time_ parsing of long sentences is still hard. So, when parsing large corpora we usually apply time and space limits (which is easy to do in Sicstus Prolog, with good recovery).

Thanks for the link to the paper!

</comment></comment></comment></comment></comment></comment><comment><USER13> This is exactly the sort od thing that makes a trip to HN worthwhile! Very cool, Thanks to high-school latin I was able to make a bit of sense of the table of parts of speech. Shame I haven't got a use for it ATM.

</comment><comment><USER12> Darn, now I'm tempted to write a pure javascript POS tagger just to show that you don't really need anything server-side (or maybe just little bits here and there so the web page doesn't need to load a 20MB model right away - <USER3> computational effort, in any case, is not so bad that you couldn't do it in JS).

Hmm. Maybe sometime.

EDIT: The Stanford POS tagger is more complex and quite a bit slower than anything you'd do on your own. To quantify this, there's methods that are 10x as fast while sacrificing 0.05%-0.2% accuracy. (Or the easy ones that are 100x as fast, but are 1% less accurate - these would be fun to do in JS).

<comment><USER14> Bold statements like that need to be backed up with code!

<comment><USER12> I don't have a JS version, but here's a functioning HMM tagger in under 300 lines of Python: <http://gist.github.com/503784>

(model loading doesn't work yet for some reason, but you see what it's doing in principle).

This uses a smoothed trigram HMM, so it should, in principle, be a bit better than NLTK's HMM tagger but not as good as serious POS tagging packages (e.g. hunpos, or the Stanford POS tagger)

</comment></comment></comment><comment><USER15> Interesting, but I wonder how commercial use of the API requires the license:

<http://otlportal.stanford.edu/techfinder/technology/ID=24472>

<comment><USER16> From <http://nlp.stanford.edu/software/tagger.shtml>

_The tagger is licensed under the GNU General Public License (v2 or later)._

The GPL, like other licenses meeting the 'open source definition', has no restrictions on use -- only on proprietary distribution under nonfree licenses.

</comment></comment><comment><USER8> lots chatter about MorphAdorner recently (relatively)

<http://dhigger.blogspot.com/2009/08/research-project.html>

[http://workproduct.wordpress.com/2009/01/27/evaluating- pos-t...](http://workproduct.wordpress.com/2009/01/27/evaluating-pos-taggers- conclusions/)

</comment><comment><USER17> It doesn't seem to be handling multi-word proper nouns as I thought it would:

"Pride and Prejudice is a good book." becomes "Pride/NNP and/CC Prejudice/NNP is/VBZ a/DT good/JJ book/NN ./." I would have thought "Pride and Prejudice" would be lumped together.

<comment><USER5> It's a little confusing but this looks like a front-end to Stanford's Part-of- Speech tagger. POS taggers do not group multi-word tokens. This would be the role of a chunker or a parser.

</comment></comment><comment><USER18> Did you develop the underlying POS tagger or the web interface?

</comment><comment><USER19> Shouldn't 'fuck' be a verb in this context?

go/VB fuck/NN yourself/PRP

<comment><USER11> Using my own tagger[1], trained using the Brown corpus:

go/VB fuck/VB yourself/PPL

It's very much in the amount and kind of training data and features used (assuming that the methodology is sound).

[1] <http://github.com/langkit/citar></comment></comment>


Show HN: Language games – simple games made with word vectors - <AUTHOR> https://github.com/Hellisotherpeople/Language-games <comment><USER2> I wonder if there are any good sets of word vectors in languages other than English (I'm specifically interested in Polish).

Same with any other language models and NLP related stuff. It's hard to find anything valuable.

<comment><AUTHOR> Yeah! Fasttext has some models in polish which I can convert into the magnitude format for you! File an issue for it and I'll get it done

<comment><USER3> Yes I would look to fasttext first, they have a huge library of vectors here: [https://fasttext.cc/docs/en/crawl- vectors.html](https://fasttext.cc/docs/en/crawl-vectors.html)

Actually facebook research are pretty good at gathering things, their ParlAI project also gathers together other sources for conversational stuff.

</comment></comment></comment><comment><USER4> It's interesting. There has been lots of ways to explore Word2Vec, but maybe this helps people get an intuitive understanding of the strengths and weaknesses better. (hint, the farther the words are from the target, the more meaningless the distance comparison becomes).

</comment><comment><USER5> "Competative" word guessing?

(And it's "pedant", not "pendant").

<comment><USER6> Yep, like most projects that use embeddings it doesn't seem that smart when you look closely. It does "better than chance" but that's not a high standard!

</comment></comment><comment><USER7> Nice project!

I did something similar when I was still at school. Scripts which emulated Codenames [1], and providing hints for given set of words [2].

I believe that is possible extension to your project - emulate Codenames games:)

[1] [https://boardgamegeek.com/boardgame/178900/codenames](https://boardgamegeek.com/boardgame/178900/codenames)

[2] [https://github.com/johnnyheineken/codenames- AI](https://github.com/johnnyheineken/codenames-AI)

<comment><USER8> I've seen your Codenames work project — it's neat!

I've also played around with word vector games — Robot Mind Meld [1] has you and a robot working together to converge on the same word.

[1] [http://robotmindmeld.com](http://robotmindmeld.com)

</comment></comment><comment><USER9> I'm failing to understand game 4:

        Player 0 it is your turn!
        The word you are trying to match in meaning is:
        antepenultimate
        Your char list is:
        ['i', 'g', 'j', 'u', 'b']
        Please input a valid word made from some or all of the provided characters

What's the right answer?

<comment><USER4> Importantly: there was no "right answer" in mind when it created the question. The letters were random, as was the word.

It calculates the distance in vector space from your word to the given word.

I think it helps if you understand what Word2Vec is, as then it becomes clear what is going on.

<comment><USER3> so you didn't have a word near penultimate and then pick some letters from it? It's really just open-ended what the users should choose? That does make it a lot harder...

</comment></comment></comment><comment><USER3> btw what did you like most about using the magnitude format? lazy loading seems good, depending on if it takes time to load the sqlite DB at each startup. That's one of the major pains of working on anything with word embeddings is that server reloads / hot restarts take forever.

The fastText vectors (see below) also use subword embeddings so I think they had potentially better results. I used FT for some Chinese stuff and I think it worked better for that since chinese _characters_ are so much more important than latin scripts.

Seems magnitude python API also has some other features like POS tagging, I wonder how that compares to say spaCy.

</comment><comment><USER3> these are neat, it would be fun to put a reactJS frontend on some of them!</comment>


How to process a million songs in 20 minutes - <AUTHOR> http://musicmachinery.com/2011/09/04/how-to-process-a-million-songs-in-20-minutes/

<comment><USER2> It would be interesting to know how long it takes to run locally on a single instance

<comment><USER3> agreed, actually, I want to know how long it takes to run on say... an i7, I've gotten pretty poor performance out of those amazon small instances when doing alot of cpu oriented tasks.

<comment><USER4> Let's pessimistically assume you have to read all information associated with each file from disk. I suspect this sort of query is IO bound since processing is minimal and assuming we can mostly avoid fragmentation on disk we could feasibly read data at 100MB/s with a consumer level magnetic drive. The article says the data set is 300GB so we could feasibly get this done in an hour.

</comment></comment></comment><comment><USER5> So... MapReduce? Kinda figured that when you had 6 zeros after your first zero.

This looks like a fun project, but I can't help but feel like Hadoop experience reports are a little late to the party at this point. Is there anyone out there who doesn't immediately think MapReduce when they see numbers at scale like this? If anything, the tool is _overused_ , not neglected.

<comment><USER6> Its a well-written tutorial on using mapreduce to process a specific large music dataset. Now someone can jump right in and start writing their own processing code without worrying about all the boring details.

Maybe the headline is a bit misleading but I find the post interesting and valuable.

</comment></comment><comment><USER7> Can anyone clarify if song data is dense? If it is dense, I am not even sure if Mapreduce is the right paradigm to use mainly because you will eventually get to a situation where transfer time overwhelms compute time.

<comment><AUTHOR> the EN song data is dense in the sense that there is far more "columns" than rows in almost any bulk analysis -- average song unpacks to ~2000 segments, each with ~30 coefficients + global features.

however, in paul's case here he's really just using MR as a quick way to do a parallel computation on many machines. There's no reduce step, it's just taking a single average from each individual song and not correlating anything or using any inter-song statistics.

</comment></comment><comment><USER8> Can you dynamically adjust the number of EC2 instances to optimise for processing time or price?

<comment><USER9> yes, the number and size of EC2 instances can be set by switches on the command line when you launch the job.

<comment><USER8> I saw that but I mean how do you know you need 100 instances? Is there a way to estimate the optimum number or at least set boundaries, such as max price or max processing time?

Do you get back info such as how long each instance took to do its job?

<comment><USER10> Since it scales somewhat close to linear, you can just calculate it yourself.

And since you pay per minute on EC2, you can either chose to have 10 instances calculate a minute or 1 instance for 10 minutes. Either way, you pay for 10 minutes of processing time.

p.s. simplified but probably somewhere in a reasonable ballpark

<comment><USER11> It's actually rounded up to complete hours, so 10 instances for a minute is 10x the price of 1 instance for 10 minutes.

For fast iterative development this isn't an issue - you can re-use the job flow (the instances you spun up), so you can launch the job several times over the course of that hour.</comment></comment></comment></comment></comment>


List of European VC Funds - <AUTHOR> http://thebln.com/2009/04/list-of-european-vc-funds/ <comment><USER2> It's missing a couple of big ones.

1\. Atomico: Skype founders VC 2\. ASI (Ambient Sound Investments): The early engineers of Skype (btw: check out their portfolio, interesting companies)</comment>


Dice-O-Matic hopper and elevator - <AUTHOR> http://gamesbyemail.com/news/diceomatic <comment><AUTHOR> See it in action at [https://www.youtube.com/watch?v=7n8LNxGbZbs](https://www.youtube.com/watch?v=7n8LNxGbZbs). Better than LavaRand.</comment>