Why U.S. Infrastructure Costs So Much - <AUTHOR> http://www.bloomberg.com/view/articles/2016-04-08/why-u-s-infrastructure-costs-so-much <comment><USER2> I don't have any love for NYC construction unions and their ridiculous work rules, but the fact that France -- France! where strikes shutting down the whole country is a national pastime -- does better means that can't be the number one answer. Similarly we can rule out corruption because even Chicago is a piker compared to Shanghai. I think both unions and corruption play a part but not of the first rank.

The first candidate I'd propose for the top slot is diffusion of responsibility stemming from a US obsession with localism. An MTA project might well be paid for by: the federal government, NY state, NYC, and bonds to be paid back by a dedicated tax in 10 downstate counties. There are contracting rules imposed by each of those sources. And it is administered by the MTA board, a monstrosity with appointments from eight different sources. The buck stops nowhere.

The second candidate is, paradoxically, the progressive era procurement rules designed to rein in corruption. They leave too little discretion. In a two sided competitive game, if one side is hampered in by slow changing inflexible rules the other side is going to dominate. That's exactly what happens. Government contractors play the system like a fiddle and there's little the government negotiators even when they are supremely competent and diligent can do about it.

<comment><USER3> > The first candidate I'd propose for the top slot is diffusion of > responsibility stemming from a US obsession with localism

If it were _only_ localism, there would be localities that were fantastic at investing in their own infrastructure. Responsibilities, even funding local schools, have been nationalizing slowly over the years without an explicit change in the U.S. Constitution. So the national government has to 'hack' the system with grants and taxes to indirectly get national policies enacted. It's a corruption of the system to work around the Constitutional protection of federalism.

One solution would be to amend the Constitution to make things like education national policy once and for all. Another would be for the national government to leave things like building roads and funding schools to the local governments.

<comment><USER4> The feds could also do some easy, central infrastructure things that would reduce school and local govt funding needs in the first place. If the schools or localities want to buy their own, great. Three examples:

1\. It seems every school uses a mishmash of Moodle, Schoology, and some Sharepoint abortion. If the feds would spend around the cost of 3 or 4 F-35 helmets ($1.6M) they could build and offer a decent school site. You know, homeworks, attendance. It could be the same software everywhere, even if you changed schools. Teachers wouldn't need retraining and kids would have their stuff follow them.

2\. Fund a decent central, unbiased curriculum with open source text books. Kansas can suck an egg. Localities really _don't_ need their own curricula. The 3 R's don't need to vary with district.

3\. Same as 1 but for local government operations software. Do they really all need their own?

<comment><USER3> On paper that's interesting. I'm skeptical that the national government can do that well, however. The rollout of the ACA websites makes it clear that the government isn't good at producing that sort of product.

Furthermore, the premise of the article is that the government is very inefficient. I'm not sure $1.6M is a fair estimate for the cost of this sort of product. There are millions of teachers in the U.S. not to speak of parents, administrators, and kids.

I'm open to providing incentives to make this sort of progress more feasible. I'm partial to better support of open source software that meets these ends (like tax deductions for time or cash contributions to OSS projects the government uses).

<comment><USER5> >The rollout of the ACA websites makes it clear that the government isn't good at producing that sort of product.

You mean that the private contractor hired to do the HealthCare.gov website, CGI Federal, did a lousy job?

<comment><USER6> Most of the things the government "produces" are not produced by the government. They're sent out for bidding by private companies - roads, guns, websites, whatever - and the government is in charge of oversight.

And the fact that they've been proven incapable at oversight and management should figure into the planning.

<comment><USER7> But what makes you so sure the government wouldn't do better taking it in- house? And even if we throw out that possibility, we're talking about implementing off-the-shelf software here, no producing custom software.

<comment><USER6> Great point! And since - as described - it's all open source, there doesn't even need to be Federal oversight or management.

If the only thing standing between now and doing it is the will and tech support, that's an easy problem to solve.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER8> We just had a novel crane/derrick start-up in New York shut down by the reigning tower-crane union [1].

I also noticed, when I lived on West 49th Street, how street-paving frequency ( _e.g._ paving a road right before putting in a water main then re-paving it right before installing Citi bikes then re-paving right before making a change to the water main...) ticked up in election years.

[1] [http://www.crainsnewyork.com/article/20160515/REAL_ESTATE/16...](http://www.crainsnewyork.com/article/20160515/REAL_ESTATE/160519910/why- is-this-little-construction-crane-illegal-in-new-york-city)

<comment><USER9> _how street-paving frequency (e.g. paving a road right before putting in a water main then re-paving it right before installing Citi bikes then re-paving right before making a change to the water main...) ticked up in election years_

Houston has the same problem. This is a form of corruption.

</comment></comment><comment><USER10> I find it worrisome that the author starts from a place of advocating borrowing to support building more infrastructure. I was pleased, at least, to see that he seemed to believe that infrastructure funds needed to be invested prudently.

I don't have a lot of faith in the levels of governments to choose the right projects. The infrastructure rot we're seeing right now seems to be a result of the fact that economic growth is not what it was when everything was built in the first place.

I think it's clear the stuff that was built in the 50s, 60s, and 70s was not sustainable outside of 6% GDP growth, and what we need now is sustainable infrastructure initiatives, rather than trying to bandaid the stuff we can't afford anymore.

<comment><USER11> The benefits for the infrastructure accrue to citizens in the future, so its sensible to borrow money (at the very low rates that the government can obtain) and pay for the benefits as they are enjoyed. If the additional cost of the interest makes the project not worthwhile, then this means the project shouldn't be undertaken, not that it should be paid for upfront.

<comment><USER10> Agreed. I won't discount the benefits of infrastructure now, pay later, so long as the infrastructure being built is economically maintainable later.

</comment></comment></comment><comment><USER12> I think this is similar to the issue that Americans tend to pay relatively high taxes and get relatively little in return, compared to citizens of other countries. I don't know to what extent this is a result of a few folks gaming the system to their own advantage (e.g. by paying very little taxes on a lot), or to the fact that by and large America is responsible for the security and peace of the global (and hence other states need not spend as much on global issues), or to the extent that American civil service appears from the outside to be very much a handout to people who can't get honest employment, or to the extent that government contractors are experts in milking tax dollars, or what.

It completely doesn't surprise me that we spend more on less infrastructure.

<comment><USER13>
        by and large America is responsible for the security and
        peace of the global

A task it took on itself mostly unasked for and unappreciated by the rest of that globe.

<comment><USER14> The northern hemisphere was blowing up and both the Axis and Allies begged the US to intervene.

<comment><USER15> 1939 was a long time ago, everyone who made important decisions then is now dead. The world was completely different back then in technology, culture and economy. No country that has been around greater than 20 years has held a consistent foreign policy. While what you said is true, it is completley irrelevant to a discussion of modern US foreign policy - we aren't the isolationists we used to be.

<comment><USER14> Where we are today is a direct consequence of what happened in the past. The US did not have military posts in Europe or Asia prior to WW2, yet they have been there since.

Further, that is when the US _actually secured_ (that's what we are talking about, right?) peace on two continents (after being asked).

</comment></comment></comment></comment></comment><comment><USER16> I'm always amazed when there's a story about road construction in the local news, and it costs like 1.5 million dollars to repave 4 blocks of some street..how is that possible?

I had a friend have his average-sized driveway re-done with cement and it was like $60,000 which I thought was insane. Remove old cement, grade the dirt, pour new cement, smooth it out...and wait.

<comment><USER17> Finally! Here's where my extensive background in concrete construction pays off! :-D (Seriously though, my father and brother run a concrete construction business and I have worked on both residential and commercial jobs for 10+ years of my "youth")

For starters - $60k is a LOT for a driveway. My brother's company charges $10 per square foot (at the top end) to remove and replace a driveway. This assumes the ground is appropriate (drainage is acceptable, no need for fill, etc.) This is based on 4" of concrete and steel reinforcing mesh and no subgrade prep. As with anything there are "price boosters" but that's a reasonable estimate. Average driveway is maybe 20x40, so that works out to be $8 grand max. This is in suburban South Dakota, so it might be more in larger metro areas.

A highway is nothing like a driveway. As others have pointed out, the thickness of concrete alone is double or triple (probably a minimum of 8 inches - more like 10 inches) and the reinforcement is considerably more (3/4 inch steel rebar mat 16-18 inches on center). You have to also consider that the subgrade needs at least a few inches of gravel, which is fairly expensive. It doesn't stop there - once the concrete is poured, there is typically a curing agent applied on top of the surface, and then the expansion joints have to be sawed and sealed. This is just for the road surface. I'll bet this cost also has to take into account curbing and sidewalks.

I've personally worked on both kinds of jobs - residential driveways and suburban streets. It's at least 10x more work "per foot" to do a road over a driveway. It's the difference between building a weekend "hacker project" and a production ready ecommerce website.

<comment><USER18> Our highways will probably last centuries if not millenia once cars and trucks stop banging on them. The Romans would be envious.

<comment><USER17> I would guess that within about 10-20 years of non-use they would be overgrown to the point of invisibility. Joints are tooled/cut every 20 feet (I think?) and grass has a habit of growing everywhere. The individual "sections" would remain in place of course, but growing things and erosion would push the joints further and further apart every season.

IMHO - the pristine unbroken roadways would be like that for only a few years/maybe a decade.

<comment><USER18> I'm not sure about that...the highways they put into are often more than three feet deep all told.

<comment><USER17> It's highly unlikely (due to cost of materials) that there would be more than 12 inches of concrete on a highway. There would maybe be 3 feet of depth including base material (dirt, gravel, or other engineered fill types). You would only see that depth of concrete on an airport runway or where the aircraft are parked.

I can't seem to find anything definitive about US road depths ATM.

</comment></comment></comment></comment></comment></comment><comment><USER19> Well that was disappointing. I thought he was on to something regarding projects being difficult to complete because of the common law system. But nope, it's the unions! Please...

<comment><USER20> I thought it was at least going to be something interesting about US infrastructure, but it's really focused on NYC. Unsurprisingly, the US does not place getting costs down for subways to be one of its higher priorities. Anecdata, admittedly, but _I 've_ seen very little talk about controlling union costs for MTA workers in the rural midwest; those poor guys are supporting a subway system that's barely there!

Really, though, infrastructure stuff is _neat_ , and we probably should start fixing it up. The main problem is the size of the US infrastructure is outstandingly staggering. There's an article [0] that's an argument for pouring money back into the highway funds (and the like), and the cash involved is worse than astronomical. At the very least, I didn't know there were over 600000 bridges in the US... I honestly don't know where we put them all. (Are overpasses included?)

[0] [http://www.budget.senate.gov/democratic/public/index.cfm/rep...](http://www.budget.senate.gov/democratic/public/index.cfm/repairing- our-infrastructure) I don't go to senate.gov much, so I'm guessing this is pretty partisan, but there's a crapton of followup links that are at least interesting.

<comment><USER19> On that note about bridges; When I was in Cuba, the tour guide made a special note to point out this bridge they had that connected one part of the island to another over a large valley. Apparently it was a big deal to get this bridge built? The Russians built it for them in the 80s, but before that, it was a nightmare moving around. It didn't look like much to me, a regular bridge that I've seen a thousand times here in Canada.

It put in perspective for me how expensive bridges really are. They couldn't afford to build _one_ , crucial bridge. It took a major super power to do it.

<comment><USER21> I pulled the data and superficially a random square mile of Cuba is about 100x more likely to get hit with a mag 7 earthquake than a similar square in Canada gets hit by a mag 6. Cuba gets spanked every two or three decades with a serious mag 7 or so.

Your 80s bridge survived the '92 quake, assuming you saw it after '92 of course. That was a biggie.

Earthquake proof bridges are no joke to build.

</comment></comment></comment></comment><comment><USER22> It's not just the US that is expensive and riddled with cost explosions. This seems to be a failure of pretty much every political system - for example, Germany, where construction costs explode (BER, Stuttgart 21, Hamburg Opera House).

The most common cause why costs have exploded is that, unlike "old times", where either the state itself has employed all kinds of construction planners and workers or awarded a specific company all construction projects in a given timeframe, now everything has to be available for bidding - usually the lowest bid recieves the contract. This, in turn, creates incentives (often lined with kickbacks for the responsible politicians!) for the construction companies to underestimate the bids and charging the government for each and every change to the contract, to cut corners and generally mismanage the projects.

Also, in the EU, projects above a certain threshold have to be advertised in the entire EU - leading to projects with little to no "native" people employed on site, and regular language problems and misunderstandings due to different legal requirements in the EU states (e.g. training of the work force, different specifications - what may very well be an acceptable project under Polish regulations is totally unacceptable by German standards).

Furthermore, regulatory oversight has been on a massive decline as competent people leave or don't even enter state service, so that too few people in government employment are available to properly assess and control construction process.

<comment><USER23> Well, maybe Europe built itself with government projects, but apart from some dubious transcontinental rail lines, America's "original" heavy and light rail systems were built by combinations of corporate, municipal and state entities. Roads were municipal matters, heavy trucks were seldom seen as that channel grew in symbiosis with the Eisenhower Highway system. Heavy trucks drive the need for durable roads, and they just weren't the norm until the Fed subsidized the crap out of inter-city highways and regulated the rail (most predominantly: the Pennsylvania RR) until if collapsed. Light-rail had its own similar history with roads and busses (and the fight for "right-of-way" at street level).

<comment><USER22> Weren't the beginnings of cross-US rail lines laid back at a time where regulatory frameworks (esp. cross-county/state, and noise emission) weren't that strict as today?

Also, back in the time it was customary to create stock companies where people bought shares to actually BUILD something of value?

When was the last time that a stock company was founded to build long lasting infrastructure?

Stock companies, these days, in my opinion focus WAY too much on creating short-term gains on the back of employees and unsuspecting investors...

<comment><USER24> > Weren't the beginnings of cross-US rail lines laid back at a time where > regulatory frameworks (esp. cross-county/state, and noise emission) weren't > that strict as today?

The major cross-US rail lines were laid down when government (IIRC, both federal and state governments) were handing out money like candy for building railroads. It was no more free market than today, its just the orientation of the government involvement was different.

And a lot of the companies created at that time in the industry were mostly created to funnel money from those subsidies into the hands of investors.

You really shouldn't idealize the early cross-US rail period either as one of an absence of regulatory involvement or one of noble value building vs. short- term extraction by capitalists.

Sure, even with all the outright corruption, rail lines to nowhere, and other abuses, there _also_ happened to get some useful, long-lasting infrastructure and organizations with long-term value built. But its wrong to overly romanticize the process that went into that.

<comment><USER25> For more details on transcontinental railroads financing:

[https://en.wikipedia.org/wiki/First_Transcontinental_Railroa...](https://en.wikipedia.org/wiki/First_Transcontinental_Railroad)

</comment></comment></comment></comment></comment><comment><USER26> Trickle-down contracting -- the many guys on top rake in big $$$, while the guy on the bottom who actually builds the thing gets a few cents.

<comment><USER27> This. Costs go x10 because of layers of subcontractors and machinery rentals up to the smallest screwdrivers.

</comment></comment><comment><USER28> In general, we _do_ make the worst deals. I know this place hates Trump, but that statement is not false.

The major issue of having a business-driven economy is that the best (or most "tactical") businessmen are running businesses, not the government.

As it stands right now, there is very little incentive for US government officials to run things efficiently. Whether they're promoted for incompetence, applauded for employing so many government employees, or getting paid by dirty contractors, it all leads towards the corrupt situation at hand.

I don't know the solution, but true campaign finance reform, removing corporate money from politics, would be the best blanket to start with. Good luck with all that, though...

Take a look at how Hong Kong's rail system operates, that's the best model I've seen.

<comment><USER29> There's also an argument that a mandate to select the lowest bid massively increases cost overruns, even when everyone in government is sincere and not corrupt.

If you're selecting a crappy company that lied about their bid over a competent, accurate higher bidder, you tend to get massive cost overruns that outweigh the promised savings. Everything looks good up front ("we'll build a new railway for only $XXX,000,000!") but in practice it's a disaster.

<comment><USER28> That falls under the category of "bad deals" too.

<comment><USER29> It does, I don't disagree, but it seems worth separating "bad deals we actively picked" from "bad deals our laws forced us to make".

<comment><USER28> Correct, and leads me back to my original point. Our nation's best minds are simply not in politics - they're in business, engineering, finance, or are retired.

Anyone even _suggesting_ that we take the lowest bid by default is unfit for office.

<comment><USER30> Except that is exactly what the law says they must do in some cases. In some cases it is left up to discretion, or we are allowed to choose a U.S. company bid over a foreign company, but government procurement does not work the same as private enterprise.

OTOH, the scale we're talking about is also _vastly_ different, and the companies involved are ultimately responsible to the taxpayers rather than just being responsible to their shareholders so there is more overhead and paperwork as well.

</comment></comment></comment></comment></comment></comment><comment><USER31> It's simple: there is so much money in our political system (and there always has been) that unless you're donating to a candidate / party machine, your pet projects don't get done. This means a lot of things that matter to average citizens like roads, schools, parks, etc. don't have anyone championing them -- unless there are investors who want to build up the area and want to externalize some of their costs.

So they make a small investment in getting some friendly candidates elected to city council or smaller county positions to ensure the access to the various groups they need to lobby to get their projects approved. In return for this investment, they get 100x in public money.

What it also means is that unless your project (or collective projects as often the case in gentrifying neighborhoods) promises a big return, it's not gonna get done. What's more is that so much of our infrastructure is so old and in such poor condition that it's often cheaper to just build new stuff in another area than it is to rip out what's already there and replace it. In Philadelphia where I live, large parts of our water and sewer infrastructure date back to the early 1800s.

</comment><comment><USER32> Usually question "Why X is so expensive in US?" can be answered with "Because such and such for-profit entity wedged itself between buyer and a seller and siphoned so much money that can buy off politicians that in other circumstances would kick it out."

I guess case with infrastructure is more complex.

</comment><comment><USER33> One thing people don't realize is that Union affiliation is actually still quite high in New York state--25%, and probably higher in the NYC area.

New York YIMBY address the topic two years ago: [http://newyorkyimby.com/2014/09/the-mta-wants-second- avenue-...](http://newyorkyimby.com/2014/09/the-mta-wants-second-avenue- subway-downpayment-but-cant-tell-us-how-much-itll-cost.html)

In it, they cite union rules that require _four times_ the necessary personnel on underground construction projects. They don't blame it all on unions, but I'm definitely in the camp that unions do much more harm than good.

<comment><USER34> It's hard to blame unions in the US for any substantial part of the cost when areas with much cheaper infrastructure costs have much stronger unions and equivalent salary levels.

<comment><USER33> My point is that New York HAS strong unions, especially for the MTA. And regardless of their salaries (which are higher than their European and Asian counterparts), they have to employ 4x more people to complete a similar job.

</comment></comment></comment><comment><USER35> With things like subway stations it's not easy to compare the cost because more 'extravagant' design will drive the cost up.

Apples and apples type comparison would be comparing things like cost of building one mile of the highway.

If you look at those figures they are very comparable with US having a slight edge over the EU - $6-10 mil per mile vs. €7-12 mil per km.

</comment><comment><USER36> This is a serious issue, and I think the author does a disservice to it by focusing on unions as the problem. At their very worst, they are only a problem for operations costs, which rarely explain why infrastructure doesn't get built. It is true that public service unions do have disproportionate political power, but they are also rarely the ones building the infrastructure, because construction is always up for bid by private contractors. And even if public unions _were_ building these things, labor typically only accounts for 20-30% of construction costs, so they couldn't possibly account for much of the 2-5x increase in costs compared to Spain or France.

The real issues at hand are bad planning, extravagant station designs, nominally-environmental NIMBY blockades, and plain old backwards procurement rules that affect all levels of US government. Alon Levy might claim that he doesn't know the answers, but I've been following his blog since he started it and it makes it pretty clear how those things contribute. Hell, the extravagance of the stations alone probably account for 30% of the difference in costs. Nobody seems to blink an eye at massive underground stations with huge art installations and expensive tile facades...yet they end up costing 10-20x more than basic stations that serve their purpose just fine.

This is a severe problem for how much we build too. Those on the left like to blame blame conservatives for not wanting to build transit infrastructure, but one of the most conservative regions in the country serves as a counter- example: the Salt Lake Valley. Sure, SLC itself is slightly blue, but step outside the city itself by a mile and you're in some of the reddest land you'll find in the US. And yet those outlying conservative areas have been throwing their votes and money at UTA to build more and more. And if you look at what they're voting on, it starts to make sense. They are building a very low cost, utilitarian, at-grade light rail line. They are optimizing signal priority and grade protection so they can get about 90% of the speed that a subway would give. Their stations are nothing more than simple concrete platforms. Simply put, they are getting a ton of transit for their money.

Compare that with Seattle, one of the most liberal cities in the US. We're likely going to see a new transit initiative soon, and it is going to be voted down just like every other transit initiative in Seattle since the 50s. It's gotten so bad than transit activists are worried that they won't even have majority support within Seattle city limits. Gee, maybe the fact that it is going to cost $50B and take 40 years to build has something to do with it. I'm fairly liberal and inclined to vote for transit in general, but I just can't justify paying hugely increased taxes for 40 years so I can finally have a subway stop in my neighborhood once I retire.

<comment><USER37> It's crazy how overdone the new Silver line is in D.C. It rides on a massive raised concrete platform: [https://www.google.com/search?biw=1188&bih=726&tbm=isch&sa=1...](https://www.google.com/search?biw=1188&bih=726&tbm=isch&sa=1&q=silver+line+dc+tysons&oq=silver+line+dc+tysons&gs_l=img.3...20436.21283.0.214188.8.131.52.0.0.0.84.4126.96.36.199....0...1c.1.64.img..0.1.81...0j0i30j0i8i30j0i24.S1bllHj56LY#imgrc=8eiwrhkB9mCvJM%3A).

This is the rickety sort of elevated track that has held up just fine in Chicago for nearly a century: [http://www.chicago-l.org/trains/gallery/images/2400/cta2483....](http://www.chicago-l.org/trains/gallery/images/2400/cta2483.jpg).

<comment><USER38> The elevated steel tracks have not "held up just fine" in Chicago. They've been neglected for decades, and they recently had to shut down a major line for almost two years just to perform necessary deferred maintenance-- essentially, rebuild the line--to keep it go.

<comment><USER39> Which one?

<comment><USER40> The Red line between Cermak-Chinatown and 95th Street [1] was reconstructed over 5 months [2] in 2013.

[1] None of the reconstructed line was elevated steel structure.

[2] Not "almost two years"

<comment><USER41> Also, the Green Line between 1994 and 1996.

<comment><USER39> I was wondering if they were referring to the Green Line stuff, which I remember from that weird HODAR/LEJACK PR campaign CTA ran in the '90s.

</comment></comment></comment></comment></comment></comment></comment><comment><USER42> I wonder how much of it goes back to the attitudes of decision makers.

A theme of this essay[1] is that decisions makers will outright reject ideas they are uncomfortable with.

So say someone believes that businesses in the aggregate will, say, do the right thing rather than mercilessly and rationally responding to maximize their benefit from a contract. It's going to potentially be difficult to convince that person that the structure of bidding and contracts is important.

I agree that this is seemingly absurd and terrifying.

[1] [https://news.ycombinator.com/item?id=11577074](https://news.ycombinator.com/item?id=11577074)

</comment><comment><USER43> Generally people buy more of things that cost less per unit and I expect this would apply to track miles just as much as it applies to , e.g., video games.

<comment><USER44> It depends heavily on the product.

Video games cost a certain amount to produce and then $0.25 to copy.

A track requires a certain amount of iron or other raw materials; these raw materials are already being produced at scale. A certain amount of economy-of- scale can be gained on the track-specific overhead (amortize the cost of the track-factory, amortize the management/sales overhead) but there is a certain floor on the minimum cost for a unit of track, and that floor is much higher (relative to its base cost) than the cost of a video game.

<comment><USER43> All the facts you bring up are true but it doesn't make supply and demand less applicable to public works projects.

Recently the Green Line Extension near me was partially abandoned because estimated costs to complete it had gone up from $2 billion to $3 billion. If the project had a reasonable (by international standards) price tag there's no way it would have been canceled even if the overrun had been 100%.

</comment></comment></comment><comment><USER45> If only we had more unions, and.... the unions seem to be presenting obstacles to getting projects done.

Which is it?

<comment><USER23> Neither. Collectivism of labor is a symptom of centralization of budgets. Large companies and large departments need large workforces.

<comment><USER46> > Large companies and large departments need large workforces.

yet, google, Apple, Facebook etc all have non-unionized employees.

A large work force does not necessitate a union. What a silly argument to make.

<comment><USER23> I wasn't making the argument you are denigrating, any more than I'd make the assertion that clouds mean rain.

</comment></comment></comment></comment><comment><USER47> I thought the reason NYC cost so much to build subways in was because it's all granite bedrock, which is expensive to tunnel through, and there's a ton of spottily-documented crap already down there.

</comment><comment><USER48> We need a corollary to Betteridge's law: newspaper headlines that start with "Why..." do not answer the question.

</comment><comment><USER49> Perhaps the NYC subway is a special case in the cost-per-kilometer club but how about something more run-of-the-mill like the MA green line extension [0] which is $2-$3B for 4.3 miles of above ground track? [1]

They recently had to go back and try to slash the budget since the Fed and State have pledged $1B each and no more. Some of the reasons given for rthe explosion from $2N -> $3B were poor oversight over the bidding process [2].

To cut cost they scaled back the station designs (apparently people were complaining after some stations became little more than a platform without even shelter from the weather), kept some bridges they originally planned to replace, and shortened a walk/bike path.

Nowhere in the news do I see anyone talking about labor rates, unions, health insurance / pension, or eminent domain issues causing the cost overruns.

I've been on the board of a small condo (27 units) which did a major residing project a few years ago. The process of spec'ing, bidding, overseeing the construction was hugely involved for the huge for us but relatively tiny project. The contractor bids, which were against a very detailed 3rd party specification, had a huge spread. During the project there were significant work items which were outside the spec and ended up being negotiated to control costs. The takeaway for me is that these projects are insanely complicated and cities just don't have the expertise and dedication to tightly manage them and wring out the cost savings. The experts outmaneuver the municipality and we end up paying far too much.

What's interesting about the Green Line Extension is the nature of the funding means the first $2B is almost like free funny money but every dollar thereafter is like spilling blood because it comes from the locals. So at least the funding structure is trying to impose some limits. And lo-and-behold if the choice is for a city to cough up the extra cash versus cutting costs, they cut the cost by 1/3rd.

I think the biggest problem is cultural. The people overseeing the project are not spending every dollar like it's their own. If the cost isn't "real" to the people running the project, it completely changes how the project is run. Human nature.

[0] - [http://greenlineextension.eot.state.ma.us/docs_about.html](http://greenlineextension.eot.state.ma.us/docs_about.html)

[1] - [https://en.m.wikipedia.org/wiki/Green_Line_Extension](https://en.m.wikipedia.org/wiki/Green_Line_Extension)

[2] - [http://www.boston.com/news/local-news/2016/01/15/report- gree...](http://www.boston.com/news/local-news/2016/01/15/report-green-line- extension-budget-issues-brewed-long-before-crisis)

</comment><comment><USER50> Red tape.

</comment><comment><USER51> > New York's Metropolitan Transit Authority operates commuter trains that > require higher staffing levels -- but as Smith wrote in another 2012 > Bloomberg View piece, those higher staffing requirements are the result more > of union rules than of necessity.

This is also true for construction. Anyone who's seen construction in progress in the US knows that there are usually 5 guys standing around for every one working. Everyone one of them is getting health care, a pension and a decent salary if they are standing around or not. And they are getting the best health care and pensions of just about any group in the US.

<comment><USER52> You want a true transit authority horror story go read up on Washington DCs. The unions are not their big issue though they do cost the TA a lot of time and money, its the money games that came about because of years if not decades of mismanagement by government appointees and officials.

[https://www.washingtonpost.com/local/trafficandcommuting/met...](https://www.washingtonpost.com/local/trafficandcommuting/metro- sank-into-crisis-despite-decades-of- warnings/2016/04/24/1c4db91c-0736-11e6-a12f-ea5aed7958dc_story.html)

and its not just infrastructure. Canada privatized into a non profit entity their air traffic control to good success. We have the TSA running security in airports where if you want timely service it will cost you hours of time and nearly a hundred bucks for membership and even that is not guaranteed. The worst example of how government mismanages everything is sadly still not fixed, the VA

There is just too much up front cost in government projects that weighs down the economy but the real danger is all the deferred costs from maintenance to pensions.

</comment></comment><comment><USER53> This article seems quite specific at first – but when you dig down, it is actually rather vague.

For example, the author complains that his mother needs a license to bake food. What kind of license? What state?

Maybe a food handlers license? These are incredibly common, and very easy to obtain.

Maybe a business license? Again – most states are falling over themselves to make it easy to give you a business license.

It's hard to tell exactly what is being argued against.

Food safety is incredibly important – and food handler's licenses are easy to obtain. States need to collect taxes, that's the basic reason a business needs to get a license. Neither of these things are controversial. That's why I'm guessing the author of this article didn't go into specifics.

<comment><USER54> > Neither of these things are controversial.

I'll give plenty of controversy, because your point is fundamental to the runaway problem of licensing and certification in the US and elsewhere. You _think_ it is just _one_ (well, you list two) licenses, but then you find out you also need commercial permits for the building, you need permits for the imports of most of the ingredients, you need certification to serve certain kinds of foods (especially in many states alcohol), and by the end you do not have just _two_ licenses you need to get, you can have dozens of them.

And there is no established canonical way to find out which you need, because they vary not only by state but by _municipality_. And even if you somehow "ask" your regional better business bureau for all the licenses and certifications you need, if _they_ get them wrong, _you_ are still liable and at fault for _not knowing_.

Food safety is incredibly important. If every distributor of food needs a license, the barrier to entry by its nature either must be insanely low or a huge amount of potential productivity lost to the licensing process. If the barrier to entry is low, violating such a license is painless and easy. It is a meme now that restaurants will only clean up the cockroaches the week of food and safety inspections in New York, and that applies everywhere.

In the end, the _only_ safeguard against unethical business behavior is an informed consumer. And those do not come from annual state inspections where the inspector is often bought off in the first place. They come from consumers having communication channels to better inform themselves and others about what is or is not trustworthy. We would have significantly higher standards of food safety if instead of state agencies doing it restaurants sought certification from highly respected private firms that do such inspections, because then the reputation of both inspector and restaurant on the line, and consumers can dynamically create or trust any number of inspectors to meet demand. Compare that to how well trusted the state is for just about anything by anyone, and consider which would be better, aside from the lovely side effect of there being no outdated over-saturated government bureaucracy around food inspections.

<comment><USER55> " highly respected private firms "

If the financial crisis has taught us anything it's that those highly respected firms tend to get corrupted easily. See rating agencies. Who verifies that the private firm is clean?

<comment><USER54> Consumer confidence. The problem with financial institutions is a systemic and intentional policy of using law to prevent competition. You need the, again, certification nightmare to be legally _allowed_ to even make a loan, let alone a home mortgage. If those barriers to entry did not exist (and again, it is death by a thousand cuts - only Wall Street institutions have the army of lawyers to stay compliant) we might have a hundred major banks again like we did fifty years ago than the five we do now.

<comment><USER55> That still doesn't solve the problem of corrupt private institutions. A lot of people trust Yelp or the BBB but from whatever I know their ratings can be bought.

<comment><USER54> So inform other people not to trust Yelp because the ratings are bought, provide alternatives (I usually use a mix of Foursquare / Trip Advisor / Google Reviews) and tell them to tell their friends not to use Yelp because they are fraudulent.

 _Most_ techies with any news awareness know not to trust Yelp, that right there is proof it works. It is just the onus of the informed to educate those not.

</comment></comment></comment></comment></comment></comment><comment><USER56> I found Node.js a much simpler platform to use, mainly because there is no worrying about concurrency.

Because of the asynchronicity of Node, don't you explicitly have to worry about concurrency issues? I understand that with callbacks you can make things essentially linear from an execution standpoint (per connection), but I guess I'm a little confused by this statement.

<comment><USER57> Did you reply to the wrong post?</comment></comment>


Lambdas and Functions in Python - <AUTHOR> http://www.thepythoncorner.com/2018/05/lambdas-and-functions-in-python.html <comment><USER2> Python's lambda also has an interesting design flaw that anyone using it will come up against sooner or later [1]. The way it interacts with Python's scoping rules is quite broken. This hinders using it as freely as is possible in, eg, Clojure.

      >>> fs = [(lambda n: i + n) for i in range(10)]
      >>> [f(4) for f in fs]
      [13, 13, 13, 13, 13, 13, 13, 13, 13, 13]

[1] [http://math.andrej.com/2009/04/09/pythons-lambda-is- broken/](http://math.andrej.com/2009/04/09/pythons-lambda-is-broken/)

<comment><USER3> A lambda is no different in scoping rules than a regular function def. If you put a def in a loop, you'll find the same behavior.

Closures are only created for functions defined inside other functions. Thus the term non-local. They're variables that are neither local nor global.

Don't call it broken when it works as intended.

<comment><USER4> "Works as intended" but also "unintuitive to most users" is still broken. Disclaimer: I have hated python ever since falling out of love with it 8 years ago for reasons just like this one so I'm definitely biased.

<comment><USER3> Since programming is a rapidly growing field and Python is the first language for many, I suspect that Python's scoping rules are in fact intuitive to most of its users. The ones who stumble are those with intuition built by some other language.

<comment><USER4> I'd argue that's the opposite. Python was the first dynamic language I learned. The scoping rules in Python didn't bother me because I didn't know any better, not because they're intuitive or well designed.

<comment><USER3> Fair anecdote. I'll offer that in my experience teaching hundreds of students, most don't have trouble with the cascade from locals to globals to builtins, and quickly grasp that globals are looked up for each call. That closures work at all confuses the ones who are thinking carefully about local scope. I may have a biased sample of students, but I think it's decent evidence.

</comment></comment></comment></comment></comment></comment><comment><USER5> "Isn't it better?"

No!!!!!!!!

The original code looks by far the best to me. Why over-complicate?

<comment><USER6> Most of the code by volume in this post's final product (and most of the ugliness you're probably seeing in it) comes from dealing separately with the arity of different operations (`compute_operation_with_one_operand`, `compute_operation_with_two_operands`, etc) -- which retains a whole lot of copy-paste-but-change-one-small-thing-in-the-middle boilerplate.

That can be taken further and generalized out.

It's been a while since I did much Python metaprogramming-navel-gazing, but if you can't do something to automatically detect the arity of the operations' lambdas/functions, you can at least explicitly annotate the operations data with their arity -- which is already implicitly being done (as code instead of data) in `compute`'s 3 if-statements + the 3 "compute_with_N_operands" functions.

From there, you'll only have 1 case once, instead of 3 cases with each one manifesting in 2 different places. And just about half of the code disappears.

\----

edit: since adjkant already covered the "automatically detect the arity of the operations' lambdas/functions" approach to removing all the cases, ([https://news.ycombinator.com/item?id=17329772](https://news.ycombinator.com/item?id=17329772)), here is the gist of what I meant by "explicitly annotate the operations data with their arity":

        import math, operator

        class rpn_engine:
            def __init__(self):
                self.stack = []
                self.catalog = {"+":    (2, operator.add),
                                "-":    (2, operator.sub),
                                "*":    (2, operator.mul),
                                "/":    (2, operator.truediv),
                                "^2":   (1, lambda x: x * x),
                                "SQRT": (1, math.sqrt),
                                "C":    (0, self.stack.pop),
                                "AC":   (0, self.stack.clear)}

            def compute(self, operation):
                (arity, op) = self.catalog[operation]
                operands = reversed([self.stack.pop() for _ in range(arity)])
                return self.stack.push(op(*operands))

<comment><USER6> Someone replied with and then unfortunately deleted an extremely good suggestion.

Instead of each operation taking a variable number of numbers and then _either_ returning a value _or_ directly changing the state of the stack (which was also a bug in my previous gist, in pushing the result of C and AC): homogenize the types of the operations by making _all_ of them take a stack and return a stack.

You could also wrap all of the basic arithmetic functions with a higher-level function so that, eg, `add` would still not actually have to be aware of the stack:

        import math, operator

        def stackFunction(arity, function):
            return lambda stack: stack[:-arity] + [function(*stack[-arity:])]

        class rpn_engine:
            def __init__(self):
                self.stack = []
                self.catalog = {"+":    stackFunction(2, operator.add),
                                "-":    stackFunction(2, operator.sub),
                                "*":    stackFunction(2, operator.mul),
                                "/":    stackFunction(2, operator.truediv),
                                "^2":   stackFunction(1, lambda x: x * x),
                                "SQRT": stackFunction(1, math.sqrt),
                                "C":    lambda stack: stack[:-1],
                                "AC":   lambda stack: []}

            def compute(self, operation):
                self.stack = self.catalog[operation](self.stack)

            def push(self, number):
                self.stack.append(number)

</comment></comment></comment><comment><USER7> This code can look a lot better by replacing the compute function with this:

[https://pastebin.com/fMtc8Git](https://pastebin.com/fMtc8Git)

<comment><USER8> It looks like that reverses the arguments? You could do

        args = self.stack[-n:]
        del self.stack[-n:]
        self.push(function_requested(*args))

or maybe

        self.stack[-n:] = [function_requested(*args)]

<comment><USER7> Yeah oops coded that up quick. Either way you fix it, it's far better than a function for calling each number of args and keeps the code readable.

<comment><USER7> For fun, an updated version that cleans up other things as well as the ordering, not relying on signatures of lambdas, etc.

[https://pastebin.com/HF41J8KQ](https://pastebin.com/HF41J8KQ)

</comment></comment></comment></comment><comment><USER9> It's interesting to note that a dictionary ("catalog") of values is essentially the same concept as an object. This is clearer in Javascript, where `{...}` notation is called an "object" rather than a "dictionary" and the `.foo` and `["foo"]` notations are mostly interchangable.

Storing functions in a dictionary is _very_ close to storing methods in an object. The only differences are sharing state via a `self` (or `this`) argument, and inheritance chains (either from classes or prototypes).

Also interesting to note that many of the author's functions, like `lambda x, y: x + y`, only make sense because the underlying language (Python) thought that constructs like `+` shouldn't be functions for some reason. Many languages seem to do this, e.g. here's a bug report I raised for PHP [https://bugs.php.net/bug.php?id=66368](https://bugs.php.net/bug.php?id=66368)

When we introduce arbitrary distinctions, like dictionary vs object, named vs anonymous functions, functions vs operators, etc. then we end up with all of the choices and complications like those shown in this article.

One of the principles I try to follow when programming is to _avoid_ making distinctions, or breaking symmetries, unless my hand is forced by the underlying dynamics. Experience has taught me that introducing distinctions in one place just introduces complication elsewhere (e.g. more cases to handle when metaprogramming).

<comment><USER10> > _Also interesting to note that many of the author 's functions, like `lambda > x, y: x + y`, only make sense because the underlying language (Python) > thought that constructs like `+` shouldn't be functions for some reason_

Actually python's + (and most operators) are available as functions, and allow for overloading as well:

      >>> import operator
      >>> operator.__add__(1,2)
      3

> _Many languages seem to do this_

"Many"? Unless you ever only touched Smalltalk, Lisp or some fringe languages, this is the norm for most languages -- operators are distinct from functions (same as expressions are different than statements). Whether this is a good idea or not, I wont go into, but it's not like this is something surprising, or something that merely "many" languages seem to do. Not just many, but almost ALL mainstream languages (C, C++, Java, C#, etc) do it.

<comment><USER11> Lisp operators are distinct from functions. It's just that arithmetic operations are functions; Lisp dialects use operators for various other constructs that cannot follow the function call evaluation rules.

C++ sort of makes operators functions. Any overloaded operator is a function. If _a_ is a class object then _a + b_ can be spelled otherwise as _a.operator+(b)_. This doesn't work for the built-in definitions of the operator, as in 2.operator+(2).

The built-in + operator in C++ can also do things that used to be impossible in C++ until its template system became more powerful. Namely, the result type of the built-in + is deduced from both operands and can be different from both of them. If C++ were being redesigned clean-slate, it would likely be that operator overloading is allowed for the non-class types like _int_ , and all the built-in operators are defined in terms of templates, whose deduction/instantiation reproduces all the rules.

<comment> cup-of-tea The thing that makes it beautiful in Lisp is multiple dispatch. In higher order functions, + is still just +, not some function that's already bound to the left hand side.

</comment></comment></comment></comment><comment><USER12> Has anyone ever experienced a time where assigning member variables on a function made sense?

The thing I love about python is the freedom to do whatever I want and the responsibility to almost never do it. I crave examples of exceptional cases.

<comment><USER13> The most common cases are decorator functions, e.g. one that counts the number of invocations: [https://stackoverflow.com/questions/44968004/python- decorato...](https://stackoverflow.com/questions/44968004/python-decorators- count-function-call)

</comment></comment><comment><USER13> That code seems suspect. Throwing and catching BaseException is bad/has some weird consequences and probably unintentional - I assume the author meant to use Exception? Pathologically avoiding `elif`. Misspelling `inspect` in the code examples which are ostensibly verbatim Python sessions.

In general, lambda functions are less necessary in Python because functions are first-class. The lambda syntax is useful for extremely trivial functions, e.g. `sorted(a, key=lambda value: value.foo_bar())`, but otherwise "normal" functions work just fine/better - especially since you can define a function wherever you want. (This is how it's supposed to be. E.g. in Java or Javascript, code using many lambdas can be a royal pain to debug.)

Maybe related, I dislike the use of `map` and `filter`. Almost all real-world cases are more readable using list/dict/set comprehension or generator expressions. The only thing missing now is assignment expressions (PEP 572, [0]) so I can do a map + filter in one comprehension.

[0] [https://www.python.org/dev/peps/pep-0572/](https://www.python.org/dev/peps/pep-0572/)

<comment><AUTHOR> Hi guitarbill, the article is intended to point out how to review the original code. I figured out a bad code written by anyone else and I tried to make it better.

About the lambdas you are right, they're good just if you have trivial functions, like in other languages. If you need more "power" just use normal functions.

PS: if I mispelled "inspect" or anything else... sorry, I'm not a native speaker, I wish I could speak and write better but the truth is that my English sucks... :(

<comment><USER13> Fair enough, I wasn't sure if the review was just fiction - a good idea to base the article on.

However, I do want to refute that English skill has anything to do with the things I'm pointing out. Your English is fine, plus I don't care, as I also had to learn English as a second language. The issues are purely technical.

First, `BaseException` is introduced in the refactored versions, not in the original code - so it's hardly Peter's fault. Second, the misspelling is in code, and it stops the REPL example from working ("from inpect import signature") - the REPL code looks like it's copy & pasted from the console, but it can't be.

So "suspect" was probably the wrong word to use here, even if that code has been altered. I hope you can use the feedback to improve the article, even if the feedback is worded poorly!

<comment><AUTHOR> For the BaseException you're right.

About the mispelled word I didn't notice it... that's weird, it IS a copy and paste, probably when I pasted it I lost that line and I rewrote it?

Can't remember, I posted the original article on Medium on December 2017. However I will surely modify the article next week to address that typo, thanks!</comment></comment></comment></comment>


New Startup Sets Out to Bring Google-Style AI to the Masses - <AUTHOR> http://www.wired.com/2014/12/new-startup-sets-bring-google-style-ai-masses/

<comment><USER2> Having examples just executable from the website is invaluable, especially to excite more people. For fun:

Relatedness of Sentences (beta)

1 - not related at all

5 - an almost perfect paraphrase

Two men are taking a break from a trip on a snowy road

Two men are taking a break from a trip on a road covered by snow 4.05

Two men are taking a break from a trip on a road covered by rocks 4.13

Two men are taking a break from a trip on a road covered by mushrooms 4.23

Two men are taking a break from a trip on a road covered by hobbits 4.27

Plenty of work ahead :)

<comment><USER3> Several children are playing with the leaves

Several leaves are playing with the children : 4.26

Indeed!

<comment><USER4> Yea, I also wasn't impressed:

"a few guys are drinking water at the mall" -> "stuff" : 4.22

lol

</comment></comment></comment><comment><USER5> > "deep learning," teaching machines to recognize images and understand > natural language using software that operates a bit like the networks of > neurons in the human brain.

'understand natural language'?

Very far from it...

> I saw the movie, where the main actor's wife was so angry - but I was having > a great day

Result 99% negative...

> he was hit hard

> he played a hit single

Relatedness 4.5, from 1-5

</comment><comment><USER6> A solid group of talent, welcome to the club. I am interested to see where the "deep learning" start-ups end up in ten years time with such a wide-array of problem sets and industries.

</comment><comment><USER7> Am I the only person who is worried that AI start-up companies are going to use AI to insert "deep learning" into every pitch deck possible?

<comment><USER3> Deep learning has become a buzzword that every CEO on the planet is going to put in his/her presentations.

e.g

Deep learning will allow us to identify new opportunities and capture more value from existing markets..blah blah blah.

<comment><USER8> is this the sexy re-brand that Machine Learning has been waiting for?

(honest question.. are they the same thing?)

<comment><USER9> They are closely related but entirely different approaches. Most machine learning is relatively simple statistical models. Deep learning means ridiculously large models. Sometimes they have millions of parameters and require rooms full of GPUs running for weeks to train. But the capacity means they can learn far more complicated functions (like machine vision or language.)

<comment><USER10> Machine learning isn't an approach, its an entire discipline. Deep learning is just a specific category of implementation of a subset (neural nets) of machine learning.

They aren't entirely different approaches, considering deep learning is a form of machine learning...

</comment></comment></comment></comment></comment><comment><USER11> Having spoken to Sven a few times, I think they are targeting industries/applications more specific than their website appears :)

If anything, I would consider this to be a competitor to Context Relevant and Alchemy API instead of Clarifai.

</comment><comment><USER12> Wait til the machines rise against us</comment>


Two major research funders to bar grantees from publishing in hybrid journals - <AUTHOR> https://www.sciencemag.org/news/2018/11/win-open-access-two-major-funders-bar-grantees-publishing-hybrid-journals <comment><USER2> I just read Wellcome's announcement [1], which has this interesting bit:

> We will no longer cover the cost of OA publishing in subscription journals > ('hybrid OA'). We previously supported this model, but no longer believe > that it supports a transition to full OA.

Whereas Plan S, IIRC, explicitly stated that research funded by them could not be published in hybrid journals, Wellcome merely states that they will not cover the cost - implying that well-resourced researchers are still allowed to use other funds to publish there?

Another interesting part of their OA policy is that they require organisations they fund to commit to DORA, i.e. to work towards not judging researchers on the names of the journals they publish in, but on the inherent quality of their work. In my opinion, this is the largest problem that is currently holding back the transition to open access and many other problems in science. (Disclaimer: I work on a project that hopes to solve this problem as well.)

[1] [https://wellcome.ac.uk/news/wellcome-updating-its-open- acces...](https://wellcome.ac.uk/news/wellcome-updating-its-open-access- policy?utm_source=twitterShare)

<comment><USER3> For academia outsiders, like me: DORA is the San francisco Declaration on Research Assessment [[https://sfdora.org/](https://sfdora.org/)]

</comment></comment><comment><USER4> Currently, for "hybrid" journals, the researcher must pay a publishing fee, typically on the order of $2000-$3000. I have an independent postdoctoral fellowship, which means I have some funding, but not enough for it to make sense to pay. These fees essentially lock me out of open access publishing.

Of course, I could simply choose to publish only in open access journals (running the risk of career consequences). I hope changes like this will lead to increasingly prestigious fully open access journals. However, I have found the most practical option to be posting my papers on arXiv.

<comment><USER5> I don't know what field you're in, but in my field, nobody pays to publish, and everyone just puts pre-prints on their websites and everyone seems totally happy with that. Is pre-prints on websites or arXiv not an option for you for some reason?

<comment><USER6> >pre-prints on their websites

People don't realize how inadequate this is long-term. Hundreds of years from now when scholars look back at the early 21st century, it's going to look like the dark ages.

<comment><USER7> Institutions with pro-OA policies will either require their employees to shove stuff into their own archive, or they'll automatically crawl their staff's sites to add to the archives. Those archives are usually controlled by the same department as the university's library, people who already have some idea about preserving things for "hundreds of years from now".

In the UK the funding mechanism for (some?) academic research hinges on publication data so there's a rationale for bean counters to ensure the archive is properly funded - if you build this Open Access archive you get the data for your funding paperwork as an output. So this creates an incentive even in fields where Open Access is not normal.

(e.g. big swathes of Physics are OA, same in Computer Science, but for all I know Paleontology is a desert for open access)

</comment></comment></comment></comment><comment><USER8> Bravo, this is a huge deal, more to likely follow, it rather seems a secular shift is inevitable at least in some manner. One wonders where this will end and what it will mean for classical publishing. In my experience, incumbent industries are more powerful than we'd imagine, and I'll bet that rather than witnessing classical publishing overturned, we'll just arrive at some kind of new equilibrium.

<comment><USER9> Agreed, I hope this helps ends the period of time where research funds were funneled to publishing houses by the billions instead of being used for their intended purpose, which is to do research. The time where they provided a useful service is long over. There are enough reputable open access channels now to be able to stop depending on closed access.

I expect the next few years, most universities will stop paying the likes of Elsevier except maybe on a case by case basis for access to older articles outside of the scope of these rules.

<comment><USER10> Elsevier has revenues of $2.5 billion and profits of around $1 billion. Double that to include other for-profit journals and you're still looking at only $2 billion.

While that's certainly a significant sum of money, it is but a small fraction of science expenditures: The US (federal) government alone spends around $150 billion p. a.

The fact that the latter has shrunk by about a third over the last decades (in terms of % of GDP) leads to my point: It seems that the issue of publishing is easily grasped and evokes some sort of deep-seated notions of fairness (or lack therefore). In turn, it gets a far larger share of our collective attention and outrage than warranted by the numbers alone.

The numbers regarding Elsevier's revenue vs profit also point to another often-neglected fact: being for-profit journals and having the luxury of not having to pay authors or reviewers does not mean that nothing of value is added by Elsevier. That misconception is why it often takes people by surprise to learn that Open Access journals need to collect hefty fees from authors.

Even Elsevier's profits may not entirely represent saving to be easily realised. Because nothing says that they manage to capture 100% of the value they are adding. In fact, it's very unlikely that they do, considering nobody has ever managed to pull of such a remarkable feat of capitalism. And the problems of switching to open access journals we have seen ever since the idea came up 10+ years ago illustrate this point rather well: We can collective groan whenever publications in Nature and Science are used as a proxy for research accomplishment. But the insistence of smart people to use them as such, when (trust me here) they have heard arguments against that practice myriad times, points at the fact that such proxies are needed, and that none better have yet emerged.

Any expectation of reducing costs by switching to open access should probably be discarded.

The benefit to be gained is something different, and the movement's name hints at it. In economic terms, it's the "deadweight loss" of the general public being priced out of this market (and the less-general public that happens to work at a research institute not able to afford every single journal subscription).

This is the age-old conundrum of copyright: We'd love to let you read this, and we understand that you can't afford its current price. But if we do, anybody would claim the new, lower, price. And the new price/volume equilibrium would not cover the costs of production.

Moving to open access is therefore not much different than suggestions for a "cultural flat-rate" or a tax-funded creative industry. It's going to be interesting how it turns out.

<comment><USER2> > points at the fact that such proxies are needed, and that none better have > yet emerged.

Yes, I feel that this is the core issue: we keep relying on journal brand names as some sort of proxy for a vague notion of "academic excellence". That is what is driving these prices up disproportionately, and which results in publishers like Elsevier being able to capture so much money merely (or at least primarily) for being the owners of those brand names, rather than for actual value they are adding. This fundamental unfairness is what gets people riled up, and what also feels like an "easy" way to increase funding for research, even if "just" by less than one perfect of US federal spending. (Which still is significant.)

Of course, there is __also __the hope that by lowering the barrier to contribute to the scientific process, we can increase the pace with which we do research.

</comment></comment></comment></comment><comment><USER2> Wellcome and the Gates Foundation have long been on the forefront of the transition to Open Access, and it's good (and luckily unsurprising) to see them join the front ranks here as well.

Note that the original consortium has already expanded from the initial 11 as well Wikipedia currently lists 14.

</comment><comment><USER11> So , they 'll switch some of their major journals to open-access-only and everyone will publish there. The publishing fee remains, and honestly i don't see the progress to better peer review, like for example, publishing of the reviews along with the manuscript

<comment><USER2> The main reason for this is that the goal of Plan S is not to improve the peer review system; its goal is to make more research openly accessible, and to limit the costs of doing so (it also involves a cap on publishing fees, though an exact amount has not been specified, and apparently Wellcome will not adhere to that part of the plan).

<comment><USER11> scihub does that already tbh. And arxiv/biorxiv

<comment><USER2> Sure, but Sci-Hub is not a sustainable long-term solution, still relies on the traditional paywalled publishing system, and doesn't legally make works available under an open license. arXiv and other preprint servers are limited in their reach.

We're certainly not in Open Access utopia yet, and anything that brings that closer should be commended, in my opinion, not criticised for not achieving other goals such as improving the system of peer review.

<comment><USER11> I don't know. Open access has been advocated for 15+ years, but scihub has the most impact in everyday work. At this point i think we should push for governmental intervention that retroactively abolishes the copyrights for all existing publicly-funded works or otherwise mandates the creation of publicly- accessible copies.

<comment><USER2> Yes, Sci-Hub has been a great boon. It's still not a sustainable long-term solution though, and not only do we need to abolish copyright for existing published work (whose copyright will already eventually -but only after far too long a time- expire), but also make sure that future published work will become immediately available to access and re-use. That's what this is about. (Note that Plan S was initiated by governmental organisations, so there's the governmental intervention for you.)

</comment></comment></comment></comment></comment></comment><comment><USER12> I have a couple questions due to lack of background and/or perspective in these matters:

      - What is the new business model that is forced upon these scientific journals?

      - Also, would this move influence the quality of content in the published papers?

<comment><USER2> \- There is no new business model enforced; that's up to the journals to figure out. However, the currently dominant model is an author-pays model, where authors pay a publishing fee ("Article Processing Charges", APCs) rather than readers paying a subscription fee. These funders explicitly free up funds to pay those fees, although this plan does include a stipulation to put a cap on those fees, since they tend to rise uncontrollably. (Although according to this article, Wellcome Trust will not be applying this cap, but will keep an eye on these fees itself.)

\- There is nothing in these plans that should affect the quality of the content. Open Access journals can maintain exactly the same system of peer review as their subscription-based equivalents.</comment></comment>


How did Tesla make some of its cars travel further during Hurricane Irma? - <AUTHOR> https://www.theguardian.com/technology/shortcuts/2017/sep/11/tesla-hurricane-irma-battery-capacity <comment><USER2> The batteries were limited by software, so they simply pushed an update to remove the limit temporarily.</comment>


Transcripts Kept Secret for 60 Years Bolster Defense of Oppenheimer's Loyalty - <AUTHOR> http://www.nytimes.com/2014/10/12/us/transcripts-kept-secret-for-60-years-bolster-defense-of-oppenheimers-loyalty.html?ref=science <comment><USER3> Oppenheimer was probably lucky to have gone through this, long term, to be accused by these clowns instead of endorsed. He saw it as the sham it was, and the characterization of a "broken man" is misleading. Had many years of lecturing, writing books, and leading the Institute for Advanced study after this, with deep support from the scientific community. Although it was deeply hurtful.. he was a loyal man, a soldier basically during the war, and his country betrayed him.

His consulting contract to the AEC was expiring in December 1953, but as a political takedown he was officially accused instead of simply not renewing. Oppenheimer had the option to compromise and resign, but he declined writing Strauss: "Under the circumstances, this course of action would mean that I accept and concur in the view that I am not fit to serve this government, that I have now served for some twelve years. This I cannot do. If I were thus unworthy I could hardly have served our country as I have tried, or been the Director of our Institute in Princeton, or have spoken, as on more than one occasion I have found myself speaking, in the name our science and our country".*

He was found to be absolutely loyal, he was not even accused of violating any security regulations. But was accused of not being sufficiently enthusiastic about a _crash program_ to build a hydrogen bomb, and that this lack of enthusiasm was a sign of a character flaw.

This "trial" marked an end of extremely wide public respect for scientists. And a new era where technical and scientific achievements are to be harvested by military industrial complex, and used by right wing strategists. It put scientists in their place..

*American Prometheus - <USER2>

<comment><USER4> American Prometheus is an _excellent_ book and I highly recommend people put it on their shortlist.

</comment></comment><comment><USER5> I don't understand how the absolute power to mark any information secret is compatible with democracy. The way the constitution was designed was to try to balance powers and checks between branches. But there is no check on the executive's ability to mark things secret. In this case the secrecy seems to have just been about covering up an embarrassment. They should not have the power to cover up an embarrassment. Seems like this invented absolute power needs judicial review.

<comment><USER6> If you think the executive branch has no legitimate purpose in keeping secrets, you should study history.

<comment><USER7> I think it's pretty reasonable to say the executive branch has no legitimate purpose in keeping secrets _from select parts of the judicial and legislative branches_. That completely undermines our system of checks and balances, which seems to work quite well otherwise.

Also, there are at least two kinds of secrets to consider: "structural" ones, like "this program exists" and specific details "agent Bob is part of this program". I could see the need to keep some of the latter even from other parts of the government, but it seems the former could do with some review. That's where the problems usually come from, after all: either government programs that overreach their authority or failures swept under the rug. At the same time, they're much less of a security threat than specific details of a program.

</comment></comment></comment><comment><USER8> As an aside:

>In 1953, a former congressional aide charged in a letter to the Federal Bureau of Investigation that the celebrated physicist was a Soviet spy.

The congressional aide's name was William Liscum Borden. If Borden's views had been adopted by the US Government, it is likely that we would have had a nuclear world war[0], instead the US built its strategy on the views of Brodie[1].

[0]: Borden's strategic views [http://ethanheilman.tumblr.com/post/29405762446/there- will-b...](http://ethanheilman.tumblr.com/post/29405762446/there-will-be-no- time-a-review)

[1]: [http://en.wikipedia.org/wiki/Bernard_Brodie_(military_strate...](http://en.wikipedia.org/wiki/Bernard_Brodie_\(military_strategist\))

<comment><USER9> undermining the idea that it was all somehow a plot by evil Republicans: Borden, the scum who initiated the witch-hunt against Oppenheimer, was a protégé of Democrat Senator McMahon, who was deeply in love with hydrogen bombs. [1]

[1] [http://en.wikipedia.org/wiki/Brien_McMahon](http://en.wikipedia.org/wiki/Brien_McMahon)

<comment><USER8> Borden himself advocated for a one-world government, which is the opposite of the Republican, Bircher, and McCarthyist positions.

</comment></comment></comment><comment><USER10> The article just confirms to me the reasons that Groves chose Oppenheimer to lead the Manhattan project, Oppenheimer was able to see the bigger picture.

</comment><comment><USER11> Is any organizational structure resistant to McCarthyism?

<comment><USER12> Any organization with secrets worth keeping must try to detect spies and repel infiltrators.

It may reduce disastrous false negatives at the cost of an increase in false positives.

In the 1940s, the Soviet Union had hundreds of agents in the executive branch of the US government. KGB records opened in the last 20 years largely vindicate McCarthy (Edit: in that the threat was real; infiltration had happened; most of his targets were Soviet agents, Communists, or associated with Communists; and security risks had to be removed from positions in which they could do harm for the good of freedom-loving people everywhere).

<comment><USER13> According to

[http://en.wikipedia.org/wiki/Joseph_McCarthy](http://en.wikipedia.org/wiki/Joseph_McCarthy)

> historian John Earl Haynes concluded that, of 159 people identified on lists > used or referenced by McCarthy, evidence was substantial that nine had aided > Soviet espionage efforts.

9/159\. A 94% false positive rate.

<comment><USER12> Why don't you quote the next sentence? "He suggested that a majority of those on the lists could legitimately have been considered security risks, but that a substantial minority could not."

<comment><USER13> Because it's nearly tautological: everyone on that list got there for _some_ reason. The fact that a majority of those reasons weren't entirely made up isn't an indication of their quality as predictors of espionage.

Since we now have access to the ground truth we can compute the actual false positive rate. The great-grandparent post suggests that such an examination would vindicate McCarthy, but I have trouble sympathizing with this view. A 94% false-positive rate (96% if you use McCarthy's original figure) is bad no matter how you slice it.

<comment><USER12> McCarthy identified several agents actively engaged in espionage. KGB records identify hundreds more that he missed.

McCarthy identified many more people in sensitive positions who were security risks, many with Communist associations. Even if those people were not actively engaged in espionage, they should not have held sensitive positions.

KGB/NKVD archives reveal they had hundreds of agents in the executive branch in the 1940s. McCarthy spurred the removal of Soviet moles, Communist sympathizers, and other security risks from sensitive positions. In this, he did the US a great service, as regrettable as false accusations are.

Edit: In response to your questions, it's as if you didn't read what I just wrote. Regarding the Soviet records of infiltration, several books have been written on these records and their revelations.

<comment><USER14> I don't get why you haven't been downvoted to hell yet: removing people from their positions for the views they hold is _unamerican_.

As an entirely aside if he identified a few real spies and missed hundreds doesn't that suggest he was ineffective, to the point of actual harm?

<comment><USER12> Removing enemy agents from positions is not unAmerican.

Removing people with known security risks from sensitive positions is not unAmerican.

He missed hundreds of agents, identified several agents and many more security risks, and spurred heightened security awareness that was a very good thing at the time. Expecting him to know the identity of every Soviet agent before attempting to remove any security risks is unreasonable.

<comment><USER15> Just because some members of the American Communist party were working for the Soviets, does not mean that every member was. Guilt by association is generally not a crime under common law.

I don't know what 'unamerican' means, I know how it was used, but I don't think anyone really knows what it means. Therefore I'm bringing up common law. Whether something is unamerican or not is irrelevant. We're supposed to live in a nation of laws, not arbitrary standards.

<comment><USER16> We're talking about security risks, not guilt.

<comment><USER17> We're talking about _presumed_ security risks, where there is a model justifying its validity but where the validity has not been demonstrated.

American citizens of Japanese ancestry were labeled "security risks" during WWII and sent to internment camps.

Do you agree that they were risks and therefore should be removed from all sensitive positions? And if so, what defines "sensitive position"?

(We know, by the way, that there absolutely were spies for Japan in the US. [http://www.sccs.swarthmore.edu/users/04/sorelle/poetry/wwii/...](http://www.sccs.swarthmore.edu/users/04/sorelle/poetry/wwii/internment.html) says that 10 were found, and none were Japanese. The only one I found by name was Velvalee Dickinson. Perhaps the risk factor was actually _not_ having Japanese ancestry?)

If there was a risk, why the lack of serious sabotage, espionage, etc. on Hawaii, where 1/3rd of the population had Japanese ancestry and therefore economically infeasible to intern them? Was it only because the islands were under martial law, and if so, how does that make a difference?

Or do you agree with the Commission on Wartime Relocation and Internment of Civilians that there was, in fact, little evidence of disloyalty? Do you agree with U.S. legislation saying that the "security risk" label was actually due to "race prejudice, war hysteria, and a failure of political leadership"?

If the latter, how does one distinguish between a (supposed true) claim for being a security risk based on political party affiliation and a (demonstrably false) claim for being a security risk based on ancestry?

<comment><USER16> We're talking about people in administrative positions, not people being put in internment camps.

<comment><USER17> All I see in the history is "McCarthy identified several agents actively engaged in espionage. KGB records identify hundreds more that he missed. / McCarthy identified many more people in sensitive positions who were security risks, many with Communist associations."

I see nothing which says that "sensitive positions" is or was meant as a term limited to the executive branch or to administrative positions.

Wouldn't a non-administrative physicist on the atomic bomb project, like Theodore Hall, count as a "sensitive position"? Since I'm pretty sure that was the view back then.

Keeping people in an internment camp rather made it difficult to _have_ an administrative position, so I don't really see the difference. Well, there's a different mechanism, but the same end goal - keep Japanese/Communists out of possible sensitive positions because some people who are Japanese/Communists are the enemy. No?</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment>


Ask HN: What's the best solution for getting a domain back to your home? - <AUTHOR> Currently using no-ip.org and it works perfectly, but I can't get an SSL cert for my ownCloud instance. Bonus points if I can use LetsEncrypt to generate the cert, and even more bonus points if I can somehow use a subdomain on my currently registered one. <comment><USER2> I would buy your own domain name and the use a free DNS hosting service that supports dynamic DNS. I use Hurricane Electric's DNS and do as you describe hosting everything at my house, and use LetsEncrypt for SSL.

</comment><comment><USER3> Not clear to me what the problem is with your current setup?

Let's Encrypt is happy to issue for (almost) any Fully Qualified Domain Name in the Internet DNS, and no-ip.org is on the Public Suffix List, so you won't be sharing your rate limit with other people.

If your ISP prohibits incoming connections on ports 80 and 443, so that you can't prove control over the FQDN that way, you can use DNS as proof, check out [https://acme.sh/](https://acme.sh/) for a client that has a lot of dynamic DNS users getting Let's Encrypt certificates this way.</comment>