$40 road tolls offer insight into commuter psychology - <AUTHOR> https://ggwash.org/view/65796/the-new-i-66-tolls-offer-great-insight-into-commuter-psychology <comment><USER2> My main concern is that with tolls that high that only people with a lot of money can afford them (I make six figures but I doubt I would ever pay $40 for a toll. I would need much more money). These are also often the most influential people so they may think that everything is OK because they can afford the toll and never suffer in traffic. I think the same is happening with health care. Most people in Congress have enough money and get a nice insurance plan so they don't even understand how bad the US health system is for people in the lower incomes. They think it's perfectly fine and no need to change anything.

It's very problematic when in a society the people at the top never experience life the same way regular citizens do.

<comment><USER3> Road capacity is a scarce resource, and it should be allocated with a market mechanism like every other resource. You can then take the economic gain created by the increased efficiency and tax it to provide more public transit services.

<comment><USER4> I'm a really market-oriented guy, but I have concerns about wealth's distorting effect on markets. That is, the "cost" of an item is much less to a wealthy person than to a non-wealthy person.

Billionaire Bob can spend $100,000 on a whim, but Pauper Paul can't scrounge $10,000 for life-critical health-care. Or after a natural disaster, market theory says equilibrium prices rise to balance demand, but prices cannot rise enough so that Bob hesitates to buy all the ice for his drinks, pricing Paul out of buying ice to keep his insulin cold.

It almost feels like there needs to be some fixed currency that everyone gets equal amounts of per day and that can be exchanged for the really important things. But then some people will accumulate that currency, and we're back to wealth distortions.

<comment><USER5> Some interesting ideas on this front:

1) Using markets to allocate resources without real money: [http://review.chicagobooth.edu/economics/2016/article/why- fa...](http://review.chicagobooth.edu/economics/2016/article/why-fake-money- better-real-money-feeding-hungry)

2) Finland charges road fines on a "day-rate" that varies depending on the offender's income: [https://www.theatlantic.com/business/archive/2015/03/finland...](https://www.theatlantic.com/business/archive/2015/03/finland- home-of-the-103000-speeding-ticket/387484/)

<comment><USER2> Making fees and fines dependent on income is a good way to get incentives right. In Germany for example health insurance is based on your income, not a fixed amount like here.

<comment><USER6> I believe that health care in the US is subsidized to some degree by people with higher income.

—- Medicare payroll taxes are proportional to salary.

—- Low income people that use hospitals _generally_ don't end up paying for their surgery and care which is passed on to people that do pay their bills.

—- Big pharma offers heavy discounts to low income patients for expensive drugs resulting in higher costs for the rest of us.

—- Under the current Afforadable Care Act free/low cost insurance is provided by raising taxes on the upper income people.

I don't know what's the best way to improve the system; I just wish the costs, who was paying for it, and where the money was going was more transparent. It appears that it is intentionally designed to be inscrutable.

</comment></comment></comment></comment></comment></comment><comment><USER7> I live in Northern Virginia, and the statement "People who were totally banned from a road now could now use it at its busiest time" is false. People were already driving on I-66 solo during HOV-only hours, they were just counting on super lax enforcement and the law of probability falling in their favor for not getting caught. When riding in the metro in the mornings (which runs parallel to I-66 into DC for a time) you saw lots and lots and LOTS of solo commuters during HOV times.

I imagine now people are just switching their EZPass (toll transponder) to "HOV" and still driving solo, which I imagine is even harder to enforce.

Traffic into and out of DC to and from Northern Virginia SUCKS, it's always sucked, and it will continue to suck probably indefinitely. There need to be more roads and more bridges across the Potomac river but there's no logistically possible way to do so.

<comment><USER8> Have you ever heard of induced demand? More roads and bridges are not the answer, especially since the associated capital costs could go a long ways towards fixing Metro and thereby provide greater utility.

Northern Virginia has some of the absolute dumbest land use policies and traffic planning of anywhere in the nation. It's reaping what it's sown with traffic congestion. This is a big step in the right direction. (I live in DC, because I can't stand Virginia.)

<comment><USER9> I never understood this argument against more roads/bridges. Even if there is no improvement in latency (travel times), isn't the increase in throughput (number of people traveling) and the associated economic benefit worth it?

<comment><USER8> In addition to the counterintuitive traffic congestion stuff, you also have to consider cost. Bridges ain't cheap. Metro isn't either... but you could make a lot of improvements to Metro for the price of a bridge.

The 7000 series has fortunately been a godsend for Metro. Now it just needs a miracle for each of: power supply, rail insulators, water leaks, automated control, and the union contract/pension.

</comment></comment></comment></comment><comment><USER10> > _Which would you prefer in general: Being forbidden from doing something at > all, or allowed to do it at high cost? Under standard economic theory, the > latter should always be better. In the first scenario, you have one option: > don 't do it. In the second, you have two options, including the same option > as before._

This is not how the human mind works.

One part is obvious: would you rather starve, with no options, or starve in front of a beautiful cheesecake, just out of reach? Or worse, in front of a cake out of reach to you, but with other people feasting on it? This is not just "envy", it's loss, in the second case we're losing out and we're extremely loss adverse.

The other part is less obvious but no less real: we average the cost of options. A set of 11 plates with one of them broken is worth _less_ to buyers than a set of 10 good plates, because the average value of 1 plate in the first case is less than in the second one. This has been demonstrated many times.

When you go from one option at no cost (slow lane) to one free option + one paying option, then the perceived value/cost of the "free" option isn't free anymore, it goes _up_ because people average the cost.

(That said, tolls are probably the best possible solution; but the transition is always going to be painful).

<comment><USER11> Even fully rational people can prefer less options. According to game theory you're better off without a steering wheel in "chicken".

And while credible threats may be uncommon, removing the possibility of defection for a group of people seems like a very common potential for gain.

<comment><USER12> Another interesting example of pre-committing to less options: sending an agent to negotiate for you.

Eg the company service rep you are talking over the phone can play the 'good cop' all day long and be nice and polite; but to make any decisions they have to call their manager, who can play the 'bad cop' or defer to 'company policy'.

(Not very relevant to the road charging example here, though.)

</comment></comment></comment><comment><USER13> I don't think people are being unreasonable by being unhappy with a $40 toll. The way the article words it, solo drivers all of a sudden saw a $40 toll where they previously had no permission to drive at all.

What actually happened was that people were expecting to see a $6 toll be put in place and instead saw a $40. Imagine going to McDonald's where your friend told you that you could get a Filet o' Fish for $1 but when you get there you find out that McDonald's was charging $5. You'd be mad. You still have the option to not buy the sandwich, but it went contrary to your expectations.

So the article is correct, but I don't think they fully accounted for the fact that people had already been told to expect something else. The article mentioned it, but then goes and talks like people are being unreasonably frustrated for tolls that were completely unexpected.

<comment><USER14> _where they previously had no permission to drive at all._

I think this is where it's getting hung up. The signage and all other road markings on I-66 did not give a solo driver the overwhelming impression that they did not have "permission" to drive on those roads. Compounding this is that it was/is only during certain portions of the day.

As an example, if I go onto the Dulles or Loudoun county toll roads, it's explicitly clear with toll booths etc... that YOU ARE NOT SUPPOSED TO BE ON THIS ROAD WITHOUT PAYING. Similar around the country on other toll roads.

If all access is just as easy as any other road and there is no enforcement mechanism (save the odd example of someone you know getting a ticket) or overbearing signage (like on the 495 EZPass lanes) then naturally people would assume it's just like any other "free" highway.

I think DC has a unique driving culture around this and slugging that unless you are just a total stickler for the rules is largely a grab bag game.

<comment><USER13> Definitely unique. I've never lived in DC but visited a few times for work and was flabbergasted when I heard about slug lines (I still am). It's a bad driving situation when you'll voluntarily get into a stranger's car, and drivers will just let random strangers in their car. That doesn't happen in any of the cities I've lived in.

<comment><USER15> They do this in the SF Bay area, it's called casual carpool. Very popular in the East Bay. I used to do it for many years, both as a driver and passenger. It's all about getting past the Bay Bridge toll plaza, and the carpool lane goes way faster than the (sometimes horribly) backed up traffic. The FastTrac lanes aren't any good when it gets so backed up.

Yes there are myriad unspoken rules, and hilarious stories about when they are flagrantly violated. But over the years I've had some great rides, met some wonderful people, discovered some new music, and generally enjoyed the experience of intermixing with fellow humanity in a random but mutually beneficial way.

<comment><USER16> Can't you have that experience every day using BART?

<comment><USER15> Walking to the nearest casual carpool pickup spot is closer & faster to city for most people vs walk/walk+bus to BART.

Regarding the mixing with humanity point, I think you'd agree sharing a car is a tad more intimate than sharing a sardine-packed train car.

</comment></comment></comment></comment></comment></comment><comment><USER17> Ignoring the morals for a second, with tolls that high and difficulty identifying which vehicles qualify for a HOV exemption, I wonder how many days of tolls a single ticket is and likelihood of being caught?

Let's say a ticket is $150. If the toll is $6 then you're likely better off paying the toll as you're likely to be caught at least once every 25 days. At $40 they'd need to catch you at least once a week for you to lose money. The $150 amount is fictional, I don't know what the ticket would cost, more just a thought experiment.

In any regard, when tolls are this high, you have to consider issues like this. What works for a $4-6 toll doesn't scale to $40 unless the tickets also scale.

<comment><USER18> Do they not have e-tags, cameras and registrations in this part of the world?

I genuinely ask, since my assumption when using toll roads here is that the fine would find you pretty much as soon as you use the road...marginal errors and bugs aside.

<comment><USER19> Do you need a tag to carpool where you are? Here in MN you do not, so it makes the HOV difficult to enforce, particularly because our Supreme Court decided you cannot be ticketed by camera. You basically have to have a cop car come up beside you.

<comment><USER20> You're supposed to have it now to get on this highway. Before there was no toll on it but it was all HOV-only during certain times of day. Now it is HOV or single drivers can pay apparently up to $40 use it too. Other roads in the area have manned toll plazas which accept cash.

</comment></comment></comment></comment><comment><USER21> > "The real reason there's sticker shock is that the real cost of road > transportation is hidden from most voters. Gas tax money goes to states and > the US Department of Transportation which flows back as what seems like free > federal money to build a lot of roads. Meanwhile, every transit project has > to scrimp for funds..." "...estimated that tolls would have to be at least > $41 to pay off the costs of construction."

A lot of people take for granted the Eisenhower interstate infrastructure.

<comment><USER22> An interesting contrast is Japan, where the highway system was totally privatized, and to get almost anywhere you have to pay tolls. A 300 km (280 mi) drive can cost you 6000 yen ($50) in tolls.

Of course in Japan there are usually alternatives. Not always any cheaper though. The bullet train for the same route as above costs 9000 yen, but is 3x faster than driving.

<comment><USER23> This is fantastic, and as it should be everywhere.

</comment></comment></comment><comment><USER24> I mean, one of the problems with tolls is that it is another form of taxing that has a bigger impact the lower income you have.

Without getting into an ideological discussion, having surge pricing on tolls would contribute to a minor divide in class.

In Australia this is a current topic, as (in Sydney specifically) there was a recent controversial highway widening which has been funded by tolls and constructed by a private company, which will own the road and tolls for the next 100 years or so. This highway leads to/from the CBD as a work commute for everyone coming from the west part of the city (the poorer part usually..) and so has direct influence on people's pockets in an already high expenses city on people choosing to live further away in a cheaper part of the city.

This perhaps does not directly apply to this specific instance, but I do think it illustrates that this is slightly more of a complex issue and has more implications than the simplification of the issue the article author wrote - if you introduce a toll but allow people to use it, that does have an impact on overall equality in society, even if you can debate the merits & size of the impact.

<comment><USER25> Ive seen multiple sources say that widening roads only makes the problem worse yet in this article and in your comment it seems like the "solution" people are going for. That leaves me wondering; Are these sources not sound or are the people in charge of transport morons?

<comment><USER26> Define "makes the problem worse".

More lanes, in networking terms, improves throughput but doesn't improve latency.

More lanes moves more cars. That's fairly straightforward.

More lanes doesn't improve the time of an individual commute on the road or cut down on congestion of the road.

People have a fixed annoyance level, and congestion will increase until that annoyance level equilibrium is reached again.

The only time more lanes genuinely improves things is when the extra lanes suddenly means that the road now has more capacity than required to carry the traffic level it sees.

Normally, the cost required to add that many lanes is sufficiently prohibitive that such a situation almost never occurs.

<comment><USER27> More lanes also induce more demand. Plus there is this [https://en.wikipedia.org/wiki/Braess%27s_paradox](https://en.wikipedia.org/wiki/Braess%27s_paradox)

</comment></comment></comment></comment><comment><USER28> > To many drivers' surprise, though, the tolls during rush hour were high, > hitting $34.50 Monday morning and reaching $40 Tuesday.

Is this like "surge pricing" on tolls?

I thought most of the tolls are fixed to allow people know in advance what they must pay. If it increases based on demand, I think people will get mad.

I think the psychology, if any, need not go towards the two choices. But rather what people thought the "dynamic pricing" will be and that was $5-$6 on average.

People get mad at ride sharing companies for their "surge pricing" so this seems in the same mold.

<comment><USER29> Ride sharing surge pricing gets nasty when it's a national disaster or other major unpredicted event and the rates are sexdecoupling, because it begins to look like profiteering off human misery. But moderate surge pricing is already a fact of life on ride sharing services.

<comment><USER12> Alas, in exactly those unpredicted disaster events the surge pricing would be useful.

<comment><USER29> How do you figure? Everybody needs to be evacuated, not just the wealthy.

<comment><USER12> Wishful thinking doesn't increase the resources available.

If there's more demand than supply, some kind of rationing will kick in. Luck of the draw, queues, higher prices, ..

Surge pricing encourages more supply (ie drivers), too.

</comment></comment></comment></comment></comment><comment><USER30> > Last week, only people in carpools with two people or more, drivers with an > exempt hybrid vehicle, [...] during peak times.

> This week, [...]. Hybrid folks and Dulles users also lost their privileges, > [...]

If I bought a home on the far end of this road thinking I could buy a hybrid or electric car and use it on the toll road, and then suddenly they take that away and start charging $30 each way, I'd be really upset.

Whether the expectation that this road would remain open to hybrid cars is a _reasonable_ one is an interesting question. I'm not aware of similar changes (hybrids losing their exemption) in other road systems, but I don't exactly read transit-hawk blogs.

<comment><USER29> California was the same way, hybrids were only supposed to get HOV access for a little while. It was written into the original law. (Don't know if that ever changed)

<comment><USER31> It's supposed to expire at the end of 2018: [https://arb.ca.gov/msprog/carpool/carpool.htm](https://arb.ca.gov/msprog/carpool/carpool.htm)

Whether or not it will be extended again is an open question.

</comment></comment></comment><comment><USER32> "The government could theoretically go back to the 20th century model of taxing more and building more free transportation, but voters haven't shown much appetite for raising their taxes."

This is the author weaseling out of addressing the reality that most Americans want higher taxes on the rich. Not taxes raised on everyone, not funds raised through purchasable perks.

[http://news.gallup.com/poll/208685/majority-say-wealthy- amer...](http://news.gallup.com/poll/208685/majority-say-wealthy-americans- corporations-taxed-little.aspx)

"A significant majority of Americans believe that upper-income people and corporations pay too little in taxes, supporting other research showing that well less than half of Americans -- 38% -- agree with a proposal to cut corporate income taxes, and that the majority favor heavier taxes on the rich."

<comment><USER33> Which would make voting in a president that openly campaigned in a platform to massively cut taxes on corporations and the rich unthinkable, right?

You can often predict reasonably well what individual segments of the voting public will vote for, but they overlap with each other so divining the will of the voters as a whole is really hard. Unless you just promise everyone the earth without any regard to the cost. That can work.

</comment></comment><comment><USER34> Let's assume that overall, this does not make the road any more or less efficient ( i.e. the same # of product, i.e. people and goods, still pass through the highway stretch per hour ).

For those whose $time_saved_in_traffic > $40, they now have a more economical option.

Imagine a business that bills out labor at $100/hr and offers to pay the $40 toll + $10 extra per day if the employee can and is willing to bill another 30 minutes. Employee is better off ( they effectively spend the same amount of time "at work" plus and extra $10 and likely slightly less gas money ). Employer bills out an extra $50, assuming 20% of that is markup over labor cost, this is an extra $10 in their pocket. The government gets an extra $40 (minus whatever the base toll is) to reinvest elsewhere.

The downside is that the remaining lanes ( those who cannot justify the cost of the fastlane ), will end up paying more in slightly longer commute times if the remaining lanes are now more congested.

What society should be paying attention to is how the efficiency of the road overall improves.

If there is open ( i.e. wasted ) capacity in the HOV lane, opening HOV lane to more traffic will improve efficiency in the road overall.

If the additional revenue can be reinvested to improve efficiency, this should improve things for everyone overall.

If there are no ( or negative ) efficiency gains, then those forced to ride in the mainstream lanes will bear the net cost.

tl;dr; Focus on improving the overall efficiency of the road.

</comment><comment><USER35> I wonder if this will increase demand for "ride sharing." Anyone who takes an Uber/Lyft will automatically have at least two people in the car, allowing them to use this highway for free. I suppose this probably depends on how long the commute is and whether the ride would be <$40 (or <$40 + gas + car depreciation).

</comment><comment><USER36> [https://en.wikipedia.org/wiki/Catastrophe_theory](https://en.wikipedia.org/wiki/Catastrophe_theory) I believe in the context of these road discussions its important to understand the math behind the usage of roads. Roads become less efficient, in terms of people per hour, the more people use them. While restricting access may be unfair it also creates value by getting cars off the road faster than what would have happened otherwise.

</comment><comment><USER37> We have the best road system in the world with high quality roads over the _whole_ country. Most roads are "rain-resistant", meaning heavy rainfall does not affect road conditions negatively. We don't have any toll roads.

We do pay a relatively high tax for it though. The article is spot on that tolls are the consequence of public not willing to increase their taxes.

<comment><USER38> Where are you?

<comment><USER37> The Netherlands.

<comment><USER39> It's OK to be proud of "the best road system in the world". As you point out, you pay taxes to support it.

However, it's easier to solve certain types of transportation problems when your population density is as high as it is in The Netherlands.

Consider that 41 out of 50 US states have a larger land area than The Netherlands. Consider that only 4 out of 50 US states have a larger population than The Netherlands.

We in the USA need to build and maintain a lot of roads. And our lower population density means that mass transit isn't nearly as viable, except in the largest metropolitan areas, as it is in most of Europe.

[https://en.wikipedia.org/wiki/List_of_U.S._states_and_territ...](https://en.wikipedia.org/wiki/List_of_U.S._states_and_territories_by_area) [https://en.wikipedia.org/wiki/List_of_U.S._states_and_territ...](https://en.wikipedia.org/wiki/List_of_U.S._states_and_territories_by_population)

</comment></comment></comment></comment><comment><USER40> Even for the honest ones, I would wager you'll find that most people fit into one of three categories:

1\. The ones that won't pay at any cost for whatever reason.

2\. The ones who will pay only if it's an emergency or they're in a rush and it's the difference between being on-time or late.

3\. The ones who will pay just so they don't have to sit in traffic. Either the money is meaningless to them or they'd rather be happy than rich.

I'm reasonably frugal, generally speaking, but I'm still category 3. My sanity is worth more to me than the $40. Even if it costs me some percentage of my daily pay. I'd rather spend _all_ my money on things that remove friction from my life so I have more time for fun than spend my time dealing with one frustration after another. My happiness is worth _way_ more to me than money. I would say that a fair portion of my time and pay cheque every month is spent on removing friction from my life.

What would be interesting is _if_ (as I predict) the majority of people fall into categories 1 and 2. If too many fall into category 3, I'm going to need to rethink my strategy.

</comment><comment><USER41> Seems like toll roads are always simply rent-seeking or exclusivity-seeking behavior by entities having greater-than-average resources compared to ordinary individuals.

On private property, toll regulation according to a market seems fair enough under a free-enterprise system.

On public property, toll roads are always unfair unless the toll rate is virtually insignificant to all potential users. Any other explanation does appear deceptive, whether intentional or not.

Therefore it is usually just destructively extractive for the average users to fund roads on public property exclusively by tolls alone.

When the public tolling entity is under the jurisdiction of electable officials in a working democracy, the proper response is for a disruptive candidate to campaign on a platform of eliminating toll roads altogether. With public support this could include public purchase of private toll roads according to eminent domain if a price affordable to the citizens can not be negotiated.

Other things being equal, this type candidate should be able to prevail in cases where toll rates are particularly significant to the citizens in general.

Assuming of course free enough enterprise and democracy working well enough . . .

</comment><comment><USER14> I guess I'm at a loss here because I've lived in DC almost 6 years now and it never occurred to me that I-66 was HOV only during certain portions of the day. I know I am not alone in this because I see similar conversations happening elsewhere.

Additionally, many comments say that single traveler HOV use is _forbidden_ however I frequently see single drivers in the HOV lanes on 66 and elsewhere un-restricted. As a computer vision engineer, I know that DC doesn't have the capability to build camera systems that can infer a single/multi passenger drivers and the number of police that are 1. available and 2. can discern as much are simply not there. Are they now mailing toll charges to everyone who does not have the EZPass system?

What they effectively did was permit free use of I-66 virtually unimpeded, through lack of enforcement of the intended laws, for years and then rapidly and expensively enforce the system.

So the question I have is, how is it that they can go from a minimally (at best) enforced HOV policy to a high toll rate system effectively overnight, and expect that driver behavior will change without outrage?

It seems like it will certainly decrease the road users, IF one of two things happen: they can identify and charge (or overcharge) drivers based on occupancy (I argue they technologically can't) OR rely on self-policing of behavior from drivers.

<comment><USER42> > guess I'm at a loss here because I've lived in DC almost 6 years now and it > never occurred to me that I-66 was HOV only during certain portions of the > day.

Here is a photo of one of the HOV restriction signs:

[https://upload.wikimedia.org/wikipedia/commons/thumb/1/17/20...](https://upload.wikimedia.org/wikipedia/commons/thumb/1/17/2016-01-03_15_04_18_Entrance_to_the_eastbound_Custis_Memorial_Parkway_%28Interstate_66%29_from_northbound_Glebe_Road_%28Virginia_State_Route_120%29_in_Arlington_County%2C_Virginia.jpg/440px- thumbnail.jpg)

This sign seems perfectly clear, provided of course one has learned the acronym "HOV".

These signs (and they are large signs, the small no-truck sign in the photo above it itself about 2 feet by 2 feet) exist at every on-ramp for I-66, sometimes more than one of them.

<comment><USER43> If you're not familiar with HOV lanes then they're not clear. I recently moved to the Northern Virginia area, coming from Michigan a state without HOV/Toll Roads, I was not familiar with all this nonsense.

</comment></comment></comment><comment><USER44> > Which would you prefer in general: Being forbidden from doing something at > all, or allowed to do it at high cost?

There are plenty of government policies or state run services that prevent or completely avoid this dynamic in the name of pseudo 'fairness'. When usually it more often restricts options for more people rather than makes it more fair.

It's easy to see why from a populist perspective, look at the outrage on Twitter:

>> I don't have to use it, but it's effing outrageous. Armed robbery. $34.50 for 10 miles? New tolls on I-66 kick off with some high rates. [https://twitter.com/andreas_adriano/status/93772317073970790...](https://twitter.com/andreas_adriano/status/937723170739707904)

... he admits he never even uses it but he still finds it unfair 'highway robbery' regardless.

And it's not just a policy helping the wealthy as it would be using the service to maximal efficacy rather than below capacity. It's a net gain for average users. As there will never be a high number of people willing to pay that fee, not slowing down existing traffic. Plus the additional revenue gained could otherwise be used to improve the operation, go back into the economy via employee income or spending, etc. It's a net gain.

</comment><comment><USER45> An individual car might not be a mass transit system but a highway is (albeit inefficient). I think it's just as absurd to use demand pricing for highways as it would be to use them for subways. (I support HOV lanes though)

The main issue I see with a non-HOV toll option on an HOV road, toll lanes, etc. is that it gives the wealthy an option that others do not have. If there isn't enough commuting capacity (regardless of medium) we need to provide more. The state should not give the rich an option to make their commutes tolerable because that's easier than solving the problems (because we all know that when the rich and powerful have problems with how the state is running things that's when they change).

The wealthy have to wait on the same subway platforms as the poor. The national parks don't have a "skip the line if you're rich enough to justify the insane price" option. The wealthy should have to sit in the same traffic jams as the rest of us. Congress should have the same healthcare options as the rest of us.

The worst part about toll lanes and the other demand based pricing for necessities is that they reduce the incentive to increase supply.

If it were independently verifiable that any given implementation of surge pricing was optimized for bandwidth (people-miles over the time period surge pricing applies) rather than latency (average speed) or revenue I might feel differently but considering how pretty much every other traffic control rule turns into a revenue source of some sort (tolls on I90 in MA were only supposed to last long enough to recover the construction cost) I quite frankly don't trust any government to do it that way unless it's 100% in the open.

</comment><comment><USER46> Is everyone else missing the fact that this may be a bug in the algorithm? Read the update to the article, tolls are "surging" despite the road being almost empty.

</comment><comment><USER47> > Which would you prefer in general: Being forbidden from doing something at > all, or allowed to do it at high cost?

This is market economics at its best. If you don't like the $40 toll go back to alternative routes you were using before they introduced the toll. I also find it ironic that "free market" Republicans are suddenly stamping their feet and demanding either price controls at $6 or reverting to the old system.

<comment><USER3> It's nuts. Reasonable people can differ about the accuracy of economic predictions on the large scale, but market mechanisms are great for things like this (and surge pricing, etc).

</comment></comment><comment><USER48> I'm a guy with a long commute and a dislike of traffic jams. I've shifted my work hours to get me to work by 6:30 am at the latest. I'll take the toll route (Boston's Tobin Bridge, $1.50) or the free route (I-93) without considerating cost because I've taught myself that time is worth more than money.

But if the Tobin toll abruptly went up to $20, my loss-aversion anxiety would spike (cf. this year's Nobel Memorial winner Richard Sunstein, for behavioral economics) and my behavior would change in ways that weren't exactly rational.

This already happened on the Tobin Bridge when the tolls were first introduced. People from the neighborhoods near the Tobin used alternate routes and clogged them. The government response was wise: issue special toll tags to neighbors giving them a discount on the Tobin. The wisdom was that it countered loss-aversion with a special privilege. It worked. (Of course, this being Massachusetts, there's a small but thriving black market in those special toll tags for wiseguys wanting to save a couple of bucks a day. But so what?)

In a place like the DC area, people react to loss anxiety by attacking public projects as well as by changing their behavior, so this experiment may not last too long. I hope they can get some skilfull behavioral economics people to work with them before the whole thing gets shut down hard by fake-angry politicians.

If the goal is really to increase vehicle occupancy, the highway needs stations named for Tom and Ray Magliozzi's Russian chauffeur (Pickup Andropoff) near the exits. Getting more people into cars has proven very difficult.

Understanding this kind of peak-load pricing, and how to apply it effectively, will help all sorts of future projects in our overcrowded metroplexes.

</comment><comment><USER49> Tolls change routing decisions. If you can't gauge a toll until you are at the road because it keeps changing, that's a risk. Dynamic tolling asks people to take a risk; that will never work unless they do like Uber and fix the price at the time you leave. It's only fair that the planning risk should fall on the central traffic manager, as they have the most information and capabilities.

</comment><comment><USER50> The true cost of a trip is masked behind sales taxes, property taxes, income taxes, various state fees unrelated to roads, insurance, rent/mortgage for the cost of space for the car, the loan for the car, etc. The technology exists to toll everyone for the actual cost per mile, with higher fees for higher weight (trucks wear roads way faster than cars), and congestion. It incentivizes more investments in cycling and pedestrian lanes, would affect how and where housing is built, and how much investment is made in public transit and where it's located.

Congestion fees are used by at least Long Island Rail Road, it's roughly 50% higher during peak times. So it can be applied to public transit as well. In fact a chunk of automobile transport is public transit because the biggest cost in the whole system is the infrastructure which is publicly owned and maintained, typically without direct user fees that relate to usage. Gasoline taxes only approximate a person's usage.

</comment><comment> gm-conspiracy So, in the article, the author says (as the 2nd of 3 possible options):

 _Add lanes, which Arlington doesn 't want and there's no money for (or would require high tolls to fund); or_

Does adding lanes alleviate anything?

I have seen multiple studies indicate adding lanes does not help w/ traffic, and actually makes things worse.

Is it simply assumed that more roads = less traffic?

</comment><comment><USER51> I find these types of articles massively deceiving, if there is a simple explanation for something, a complicated explanation isn't required, ie. the human psychology. Here is one reason why the drivers reaction makes sense, when previously people were disallowed to use the toll road it wasn't about money when you charge them $40 it is all of a sudden a question of money. Previously people were perhaps understanding of congestion problems and it's natural when you have so many cars on the roads that the government should disallow some and they understood that otherwise the traffic wouldn't move. Now when you make it about money, it's no longer arbitrarily decided who gets to ride in the toll road, it's about who can afford to ride on the toll road. (or to the extend that it was arbitrary before)

</comment><comment><USER52> Consider that taxpayers fund and build airports. Available seats on an airplane are priced using demand based pricing.

Taxpayers provide land and (sometimes) fund rail infranstructure. Most long distance rail uses demand based pricing (the last seat costs more than the first).

Taxpayers also (sometimes) provide/pay for land and the costs to build roads. Until now, we haven't seen demand based pricing (the last slot costs more than the first), but it actually fits well with other modes of taxpayer funded transportation.

This approach also considers the cost and value of a road network in a congested area. This could possibly fund other new transport in the area if it really is such a valuable commodity.

</comment><comment><USER20> I understand the psychology of it as far as the high price goes, I think also people are appalled that there are others who can afford such obscene prices too, not just that the prices are so high. And well the two counties funneling traffic into this road Fairfax and Loudon are the richest counties in US, I have no doubt they'll be enough people who will pay the $40.

But what most people are upset about is that the hours when the whole highway becomes HOV-2 only have been expanded. So solo drivers who were getting up early, could have made it into the city before the restrictions kicked in, now won't be able to, and will have to funnel to secondary roads.

</comment><comment><USER53> When the price algorithm goes above some unacceptable amount, they should just change it to "Carpool only", as it was before, and there would then be no reason for people to complain.

If they made the "Carpool only" have some sort of fine for violations, say $100, paid the same way as the toll would have been, people could still use it in real emergencies.

It sounds like the calibration of the toll amount is a bit off, and can probably be adjusted over time as the systems get experience managing the demand of the road. If they'd launched with an $8 toll, and ended up with gridlock on the road, would that have been better?

<comment><USER54> With a $40 toll and a $100 violation, it pays off to break the rules if you only get caught every third day(!)

Incidentally, I've NEVER seen anyone get pulled over for driving solo on the car-pool lane.

</comment></comment><comment><USER55> There is an important point lost here. People are saying 'people who couldn't use the road before now have the option to', but they are ignoring that the restricted hours were increased by 90 minutes in the morning and evening. I used to have an option to get to work super-early, or work till 7pm then head home on a road that was no busier than the rest of the regions traffic. Those options are now off the table. What was legal, fair, free, and without congestion is now blocked by a toll that is pushing early morning traffic onto roads not built for commuter traffic.

</comment><comment><USER56> If the road was already pretty full during rush hour when no solo driver was allowed, how is this new rule going to help by adding even more drivers? Sounds like a major fail to even have that kind of rule in first place.

</comment><comment><USER8> This is a decent follow-up noting that the average toll is well below the peak toll. Of course, it doesn't talk about the median.

[https://www.washingtonpost.com/news/dr- gridlock/wp/2017/12/0...](https://www.washingtonpost.com/news/dr- gridlock/wp/2017/12/08/forget-the-infamous-40-toll-heres-what-the-i-66-tolls- are-averaging/)

</comment><comment><USER57> The article quotes a legislator:

> which is why I called charging the highest toll in the country what it is: > price gouging.

No, no, _no_ it's not price gouging: it's charging what the privilege of driving in the fast lane is worth.

The problem in northern Virginia is that there are too many people commuting. Fortunately, there's the entire rest of the country that they can move to.

'Not located in northern Virginia, New York or California' is a very real marketing edge for a potential employer.

</comment><comment><USER58> Stuck in traffic, you might as well be stuck in prison. Say your loved ones are counting on you to be somewhere. When you are stuck , you're letting them down! Suddenly you have an option when there were no options previously. Now $40 looks pretty cheap in the face of fighting a spouse, disappointed children, etc.

<comment><USER59> If I drove there twice a day I wouldn't need to go to work anymore, because it would (almost) cancel out my salary. Granted I don't work with an American salary, but they're not THAT much higher. I don't know if that's what you call cheap.

</comment></comment><comment><USER60> This is kind of a joke, but imagine this scenario:

If I illegally immigrate to USA, how much I would make just to take a ride with people?

I believe I could charge US$20 per trip. One trip early morning, go back by public transportation (it is just 10 miles!), and get another ride. The same at rush hour in the other direction. It would me about 80 * 22 = $1760 each month. Nothing bad. Better than a minimum wage of $1,150 per month.

This is a kind of job that exists in a lot of places: [https://gizmodo.com/10-transportation-related-jobs-that- only...](https://gizmodo.com/10-transportation-related-jobs-that-only-exist- in-certa-1565635360)

<comment><USER61> There are plenty of slug line commuter parking lots around the area that you can find people that will ride for free in your car so everyone can take advantage of HOV lanes. It's definitely a thing in some places where the passengers have no real need to be going anywhere in DC, people are just trying to get to work.

</comment></comment><comment><USER62> It's not about the price, it's about expectation.

The shock comes from people thinking, you've just opened a road for the rich, not for the every man. Roads have not, historically in the US, been excluded to a certain group of people. Of course people will react harshly.

Some economists need to get their heads out of their asses and live in the real world for a bit.

</comment><comment><USER63> Can it be cheaper if you drive faster ?

ie. 45mph = $40

ie. 80mph = $30

ie. 100mph = $6 (for contrast)

et cetera ? That would make traffic flow and people happy ?

Not US driver, don't know the US speed limits. On oposite side, Germany doesn't have speed limits on some tolls. [https://en.wikipedia.org/wiki/Autobahn](https://en.wikipedia.org/wiki/Autobahn)

<comment><USER64> Keep in mind that the safety distance you have to the preceding driver increases with speed. Additionally the large speed differential between cars leads to more frequent lane changes, acceleration and breaking, etc. Which hinders the flow of traffic, reducing throughput.

In Germany it's common to use electronic signs to limit speed when the highway is (moderately) congested to help traffic flow more smoothly, increasing throughput (typically to 100 or 120 km/h).

<comment><USER63> Valid points. Thank you. Was just stupid 'why not' late night thoughts.

</comment></comment></comment><comment><USER65> they should fund public transport proportionally to a tax applied to fuel.

that way more cars also means more buses (or well maintained if people insist on using cars)

every year crank up the fuel tax and you'll see more people using busses.</comment>


Introducing Bullet: The Functional PHP Micro-Framework - <AUTHOR> http://www.vancelucas.com/blog/introducing-bullet-the-functional-php-micro-framework/

<comment><USER2> A good idea for a blog post would be to compare the features of your project with Laravel and Silex, both of which also do matching and anonymous functions.

I understand yours has nesting but with all the frameworks out there, it becomes hard to choose a newer one without a comparison with the existing ones.

<comment><AUTHOR> I think a raw feature comparison wouldn't quite capture it, because nearly all micro-frameworks have essentially the same features on a list or comparison grid. I think the real "magic" of Bullet requires using it a bit to fully understand why it is a fundamentally better and more flexible approach. I have thought about making a screencast series or a nice tutorial with an example app I made with it. I am leaning towards a screencast or two because I feel a lot of the niceties of this functional approach have to be explained since most people are used to the more typical flat approach.</comment></comment>


Our V1 website of ICO launched today Tough work now at play - <AUTHOR> https://mpk.impak.eco/en <comment><USER2> Well done. Love the mission.. is there a floating VIP code / Easter Egg :)

<comment><AUTHOR> Hey! Yes indeed! Keep in touch and you'll get to discover your's</comment></comment>


Brexit Countdown - <AUTHOR> https://interactive.news.sky.com/2017/brexit-countdown/ <comment><USER2> An Englishman told me earlier that the changeover happening at midnight CET (rather than GMT) was "the one final indignity" afforded by the EU.</comment>


Let's Remove the Global Interpreter Lock - <AUTHOR> https://morepypy.blogspot.com/2017/08/lets-remove-global-interpreter-lock.html <comment><USER2> The comments here are missing a massive use case: shared memory. Shared memory isn't just about programmer convenience. It's about using a machine's memory resources more effectively.

Yes, shared memory is available in multi-processing, but it doesn't necessarily interact well with existing codes.

I've been working on adding Python support to Legion [1], a task-based runtime system for HPC. Legion wants to manage shared memory so that multiple cores don't necessarily need multiple copies of the data, when the executing tasks don't conflict (all are read-only, or access disjoint data). Legion is C++, so this mostly "just works". Some additional work is required to support GPUs, but it's still not so difficult. But with Python, if we go with multiprocessing, we have to switch to a different mechanism. Worse, Python is an optional dependency for Legion, so we can't depend on Python's multiprocessing support either.

If you have a large existing project, and a use case that can take advantage of shared memory, being forced into Python's multiprocessing scheme for parallelism is a pain.

We've been investigating using a dlmopen approach as well, based on this proof of concept [2]. Turns out that dlmopen in every available version of libc has a critical bug that prevents it from being practically useful, if you have any desire to make use of native modules. You can build a custom libc with this patch [3] but rolling a custom libc is also a massive pain.

In all likelihood we'll end up rolling our own multiprocessing to make this work. If the GIL were truly gone though, we could potentially avoid many of these issues.

[1]: [http://legion.stanford.edu/](http://legion.stanford.edu/)

[2]: [https://news.ycombinator.com/item?id=11844268](https://news.ycombinator.com/item?id=11844268)

[3]: [https://patchwork.ozlabs.org/patch/496559/](https://patchwork.ozlabs.org/patch/496559/)

<comment><USER3> ^This. It is a very common usecase for applications I work with to create a very large in memory read-only pd dataframe and then put a flask interface to operations on that dataframe using gunicorn and expose as an API. If I use async workers, the dataframe operations are bound by GIL restraints. If I use sync workers, each process needs a copy of the pd dataframe which the server cannot handle (I have never seen pre-fork shared memory work for this problem). I don't want to introduce another technology to solve this problem.

<comment><USER4> The pickling implementation of joblib has support for memory mapping numpy arrays nested in arbitrary data structures such as pandas dataframes.

Save the dataframe in a folder that can be accessed by the gunicorn worker:

        import joblib
        joblib.dump(df, '/folder/shared_data.pkl')

Then in the code run by the flask / gunicorn workers themselves:

        import joblib
        shared_df = joblib.load('/folder/shared_data.pkl', mmap_mode='r')
        # use the shared_df as usual (inplace modifications are not
        # authorized)

Some pandas function can have issues with read-only buffer though: [https://github.com/pandas-dev/pandas/issues/17192](https://github.com/pandas- dev/pandas/issues/17192) (caused by a currently unsolved bug / limitation of Cython) but it can work for your use case.

<comment><USER3> This looks very interesting. I am reading the docs [https://pythonhosted.org/joblib/parallel.html#manual- managem...](https://pythonhosted.org/joblib/parallel.html#manual-management- of-memmaped-input-data) and it looks like it would help a lot (possibly solve the issue). Do you have any experience using this in production?

<comment><USER3> DAMN. I just did a basic test and it kinnda just worked?!? I created a test dataframe of 100M rows X 10 cols which took up ~2.3G and then used joblib.dump within the on_starting hook which is run when the gunicorn master starts up. Then loaded that df in with joblib.load within the worker and the total memory consumption was practically flat. Then I bumped up the number of workers to 20 and still flat. That is actually amazing. Coolest thing I have seen in months for how easy it is. Now I have to test out if the analytics actually work and a deep dive into the mechanics of mem-mapping.

<comment><USER4> Thanks for your feedback. I am glad I could help you.

</comment></comment></comment></comment></comment></comment><comment><USER5> Having ported Ruby to IBM's Blue Gene/L my advice is to forget about the GIL. Run one Python process per core. Use something like MPI2 for message passing communication. Ruthlessly eliminate bloat code from production binaries and statically link all the things.

<comment><USER6> I agree wholeheartedly. Almost every time I hear from someone who is upset about the GIL, I find that they would be much better suited to using multiprocessing instead of multithreading.

With 80% of the developers out there, they are basically assured of producing better, more stable code this way.

<comment><USER7> Python's "multiprocessing" means launching another Python interpreter in a subprocess. Each process has a full copy of the Python environment. They may share the base interpreter, but there's a separate copy of every package loaded and all data. Memory consumption is bloated and the CPU caches thrash. Launching a subprocess is expensive; it means a full interpreter launch and a recompile/reload.

"Multiprocessing" is useful when you have a lot of work to do concurrently and not too much data to pass between processes. I've used Python subprocesses that way. Parallelizing your number crunching is probably not going to work very well.

<comment><USER8> If CPU load is an issue, why would you be using an interpreter in the first place?

<comment><USER9> A lot of the heavy lifting is done through calls to C libraries anyhow, with Python just being a convenient way to pass the data around.

<comment><USER10> Indeed, and in that case the GIL is effectively a non-issue (there's no requirement for the GIL to be held by non-python code).

<comment><USER7> No, no, if you're manipulating Python objects from C code, you have to hold the lock. You can release it only when not doing anything with objects in Python's memory space. Otherwise you get race conditions and intermittent crashes.

</comment></comment></comment></comment></comment></comment></comment><comment><USER11> I feel like the GIL is, at this point, Python's most infamous attribute. For a long time I thought it was also the biggest flaw with Python...but over time I care less and less about it.

I think the first thing to realize is that single-threaded performance is often significantly better with the GIL than without it. I think Larry Hasting's first Gilectomy talk was extremely insightful (about the GIL in general and about performance when removing the GIL):

[https://youtu.be/P3AyI_u66Bw?t=23m52s](https://youtu.be/P3AyI_u66Bw?t=23m52s)

I am not sure I would, personally, trade single-threaded performance for enabling multi-threaded applications. I view Python as a high-level rapid prototyping language that is well suited for business logic and glue code. And for that type of workload I would value single-threaded performance over support for multi-threading.

Even now, a year later, the Gilectomy project is still slightly off performance-wise (although it looks really really close :) ):

[https://youtu.be/pLqv11ScGsQ?t=27m32s](https://youtu.be/pLqv11ScGsQ?t=27m32s)

As noted elsewhere, multi-processing offers adequate parallelization for this type of logic. Also, coroutines and async libraries such as gevent and asyncio offer easily approachable event loops for maximizing single-threaded resource utilization.

It's true that multi-processing is not a replacement for multi-threading. There definitely are tasks and workloads where multi-processing and its inherent overhead make it unsuitable as a solution. But for those tasks, I question whether or not Python itself (as an interpreted, dynamically typed language) is suitable.

But that's just my $0.02. If there is a way to remove the GIL without negatively impacting single-threaded performance or sacrificing reference counting for a more robust (and heavy) GC, then I am all for it. But if there is not...I would just as soon keep the GIL.

<comment><USER12> The GIL has been a much bigger problem for perception than it ever has been for performance. Python has lost more mindshare over it than anything else. The few machine cycles that were ever saved by moving away from it were far outweighed by the waste of human cycles.

<comment><USER13> The few machine cycles that were ever saved by _NOT_ moving away from it (which is the _ONLY_ justification for keeping it) were far outweighed by the waste of human cycles.

If Python would simply suck it up and eat the 20% performance hit, we could _stop talking about the GIL_ and start optimizing code to get the 20% back.

</comment></comment></comment><comment><USER14> Could someone who really wants to get rid of the GIL explain the appeal? As far as I understand, the only time it would be useful is when you have an application that is

      1. Big enough to need concurrency

      2. Not big enough to require multiple boxes.

      3. Running in a situation that can not spare the resources for multiprocessing.

      4. You want to share memory instead of designing your workflow to handle messages or working off a queue.

#4 does sound appealing, but is it really worth the effort?

<comment><USER15> There are many cases where the objects are too big to be passed around. Python is used a huge amount in Machine learning and datascience, where being able to do parallel work on stuff already in memory would be great.

<comment><USER16> Can't this already be handled by calling out to a C/C++ or FORTRAN procedure that processes the data in multiple threads? For number crunching, Python is almost exclusively used as glue.

<comment><USER17> You CAN handle it, but why should you have to? If it's possible to remove that barrier, then it absolutely should be removed. If the only answer to a problem is "use another language", then the language in question has a limitation that needs to be addressed.

<comment><USER18> It is not a limitation at all in this case. Python is just a front to Tensorflow and similar libraries/frameworks so GIL doesn't matter there.

</comment></comment></comment></comment></comment><comment><USER19> > We estimate a total cost of $50k...

Just looking at it from a financial perspective, having a great Python interpreter that doesn't have a GIL seems like a no brainer for $50,000, and it creates another reason why people should take a look at PyPy.

Side note: if you haven't looked at PyPy, check it out, along with RPython

[https://rpython.readthedocs.io/en/latest/](https://rpython.readthedocs.io/en/latest/)

<comment><USER20> How can they estimate this? What about all the libraries that might not be compatible with the solution PyPy comes up with?

This feels like a number that might in the end blow up to 10x the original estimate.

<comment><USER21> It's not the PyPy developers' job to make every Python library threadsafe, people writing libraries will have to make their code threadsafe, like in every other language.

<comment><USER22> There is a clear difference here, though. Making a change that could lead to poorly written libraries now being broken is clearly the fault of the change. Userspace for these libraries is defined by how it is, not how it was intended.

(And really, was it intended to be dangerous in this way?)

<comment><USER23> > There is a clear difference here, though. Making a change that could lead to > poorly written libraries now being broken is clearly the fault of the > change.

No, these libraries are already semantically broken in the same way e.g. libraries which didn't properly close their files and assumed the CPython refcounting GC would wipe there asses were broken.

They're already broken under two non-GIL'd implementations.

</comment></comment></comment></comment></comment><comment><USER24> There seem to be a lot of naysayers in the comments about removing the GIL. Multiprocess parallelism isn't always appropriate, so I find this to be a very promising change that will definitely make me want to switch to PyPy. Here are the use cases I've found multiprocessing to be inappropriate:

* High-contention parallel operations. Doing synchronization through a Manager (a separate IPC-based synchronizing broker process) is of course less preferable than, say, a futex.

* Embarrassingly parallel small tasks. This is a big one. If the operation being parallelized is short, then message-passing overhead takes up more runtime than the operation itself, like a bad Amdahl's Law scenario. Shared address space multithreading solves this problem.

* Related: parallelization without the pickling headaches! Many objects can be synchronized but not easily pickled or copied! True multithreading would really enable a large amount of use cases (map a lambda instead of a named function, anyone?) since the same Python interpreter can just pass a pointer to a single shared object.

* Related: lots of libraries (Keras, TensorFlow, for instance) make heavy use of module level globals, and aren't meant to be run on multiple cores on the same machine (TF, for instance, hogs all GPU memory). Multithreading in these deep learning environments (assuming PyPy support from those packages) is useful for parallelizing the input ingestion pipeline. But this point isn't TF/Keras dependent; I can't recall other modules but don't doubt the heavy use of module-globals that's unfriendly with fork()-ing, especially if kernel-related state is involved.

<comment><USER25> > Multiprocess parallelism isn't always appropriate

Using Python isn't always appropriate.

<comment><USER24> Are you saying that because a language is missing something, when considering a fix for that thing, the existence of other languages/solutions is an argument against that fix?

<comment><USER25> I'm saying that hammers and saws exist cause more than one tool is needed to solve problems.

</comment></comment></comment></comment><comment><USER26> This seems like a good place to spruik something I made, a Python package for profiling how much the GIL is held:

[https://github.com/chrisjbillington/gil_load](https://github.com/chrisjbillington/gil_load)

In my experience, the GIL is not held for nearly as high a proportion of the time as people think it is, because properly written C extensions and blocking io always releases the GIL. So long as the proportion of time the GIL is held is not approaching 100%, then you can still get gains from threading. This is almost always the case in numerically heavy code that uses numpy or scipy, since the extensions release the GIL. Threads work almost just as well at speeding up this code as in any GIL-free interpreter.

And usually long before you consider multithreaded code, you'll want to move the bottlenecks of your code over into Cython or something, since that can give speedup factors much larger than multithreading. In which case all you need is a "with nogil:" around the the meaty bit of you Cython code, and then it too will be able to get speedups from multithreading.

</comment><comment><USER27> The ideal solution is for someone to design a new programming language that is as similar to Python as possible without requiring a global lock. Rarely used features that make it hard to parallelize Python would be dropped. STM might be built into the language instead of being hacked into one implementation, etc.

<comment><USER28> So basically recreate an entire language and library ecosystem because there is one feature that is less than ideal? I hope you realize why a better approach may be to reengineer that one component...

<comment><USER29> Python has many less-than-ideal features. Do you think we finally got it right, that we will use Python forever, and that the library work of the past decade or so is irreplaceable?

"Is it possible that software is not like anything else, that it is meant to be discarded: that the whole point is to see it as a soap bubble?" \-- Alan Perlis

<comment><USER30> Just to go from py2 to py3, which was relatively a MUCH smaller change than a whole new language, it's taken a decade and it's still far from over. I don't see how a whole new language would be any better. And it's not like there's a lack of new languages popping up left and right. There's a reason most of them just die out. It's insanely hard to gain critical mass unless you have a huge backer, like a whole organization or company using the language.

</comment></comment></comment></comment><comment><USER31> "It mostly works for simple programs, but probably segfaults on anything complicated" is not a promising beginning. Starting with race condition chaos and trying to patch your way out of it with "strategic" locking

a) Inspires much less confidence than starting with a known-correct locking model (the degenerate case being a GIL) and preserving it while improving available concurrency.

and

b) Seems at least 50/50 to end up without much in the way of tangible scalability gains once enough locking has been added to reduce the rate of crashes and data corruption to an acceptable (?!) degree. At least that was my takeaway from all the challenges Larry Hastings has documented while working on the gilectomy. Sure, they don't have to worry about locking around reference counting, but it's not like writing a (concurrent?) GC operating against concurrently executing threads isn't a significant design challenge itself with many tradeoffs to make.

<comment><USER32> > "It mostly works for simple programs, but probably segfaults on anything > complicated" is not a promising beginning.

Perhaps they would have done better to say "it works correctly for all programs that do not assume the built-in data structures are threadsafe". That is an accurate description, what you quoted is a reasonable approximation.

</comment></comment><comment><USER33> This would be great if it means we can run the C portions of Python in threads without performance hits. I recently started a little project that is a cross- platform GUI for batch bzip2 compression, and Python did it quite well with its built-in bzip2 module. But, once I tried to do it parallel, the performance impacts of GIL were obvious. Yes, you can work around that with multi-process, but I'd rather not be spamming the running processes list and have to actually handle seperate processes that should be threads.

In the end I settled for C++ and QT with the native bzip2 library with a few modifications.

<comment><USER34> > I'd rather not be spamming the running processes list and have to actually > handle seperate processes that should be threads.

I may be a bit naive asking this... but why would you care that much?

Looking at activity monitor on my Mac, I count 14 Google Chrome Helper Process instances each spawning upwards of 13 threads. Adobe does something similar, as do several other programs/applications on my machine. Yet, my machine is mostly idle.

I can only speak for myself here. If I want something done on my computer... I don't care if it spams my process list if that is what it takes to complete the task. Don't crash my machine, but do what you have to do to get it done quickly.

<comment><USER33> This is a parallel compression application that uses all cores of a system by default. On some systems, it may use 100% HDD, others near 100% CPU. Its meant to take up as much resources as it can unless its core usage is lowered. But, with any program that has a high workload, the potential exists that the programs UI will not respond, or perhaps your desktop won't even allow you to get to the UI to stop the process. This is where task manager saves the day.

Along with that, I like it to be a single process so its easily wrappable in whatever monitoring or process-throttling application you want. I will admit I'm completely assuming that multiple processes is harder than a single process to do that with.

Also, when you get up to the 16 thread count, seeing that many processes pop up at the top of your process list is both annoying and doesn't let you know how much the application overall is using easily. It could also be scary to some users who have never seen that before and think its trying to run a whole bunch of programs.

Yes, some of those are clearly nitpicks and not good technical reasons, but this is a problem that is fixed with a good framework anyways.

</comment></comment></comment><comment><USER35> I just can't stop thinking that somewhere along the line one of the Guidos should have reacted to handing out global locks left and right. I mean, as long as its only you and your friends using it. But once it starts spreading, these are the kind of issues that need to be kicked out of the way asap. Lock granularity affects the entire design of client code, reducing it basically means rewriting everything.

Ah well, at least it serves as a warning sign for budding language composers as myself. Snabel did full threading before it walked or talked:

[https://github.com/andreas-gone- wild/snackis/blob/master/sna...](https://github.com/andreas-gone- wild/snackis/blob/master/snabel.md)

And to any Pythoneers with sore toes out there: pick a better language or learn to live with it, down-voting me will do nothing to solve your problems. It's a tool, we're supposed to pick the best one for the job; not decide on one for life and defend it to death. Imagine what could happen if language communities started working together rather than competing. There is no price to be won, we're all being taken for a ride.

</comment><comment><USER36> Ick, I'd forgotten how monkeypatchable the core of Python was.

If not for that, I'd focus on supporting some kind of pseudo-process where multiple instances of the Python interpreter could be loaded but they would only share pure-functional libs which, I assume, could be used in a threadsafe fashion... but then you run into the mutability of those libs. Well, the mutability of _everything_ in python. Plus what happens if those libs expose anythign that you could hold a reference to - what happens to refcounting in a multithreaded Python?

Honestly, I feel like the world has passed Python by. At this point the cost of its performance limitations don't seem to be worth its payoff. Not that it's a bad language - I _like_ Python. I just don't really feel the need to use it for anything anymore.

</comment><comment><USER37> Excellent! Where's the Donate button or call to action for businesses who want to support this? There's a small link in the sidebar to "Donation page", but that doesn't seem to have a place to donate for the remove-the-GIL effort.

<comment><USER38> As mentioned in the blog post the individual donation buttons are not a resounding success. I'm happy to sign contracts with corporate donors (or even individuals) that we'll deliver. My mail should be public, if not #pypy on freenode or fijal at baroquesoftware.com

<comment><USER39> Is the issue that individual donations are unpredictable (and therefore difficult to use as justification for such a large scope increase)? Would you consider setting up something akin to a Patreon to allow individuals to commit to recurring monthly support for the project?

<comment><USER38> The main issue is that the effort it takes to setup and maintain it greatly outweighs the amount of money we get (typically). There is also complexity with taxation, jurisdictions and all kinds of mess that is usually very much not worth couple dollars (e.g. $7/week on gratipay for example)

</comment></comment></comment></comment><comment><USER40> Personally, I don't think the GIL matters. First of all most of us run apps on Linux which has reduced the overhead of processes so much that threads have lost much of their advantage. Secondly, people understand that locks are generally a bad thing to use unless you really are a threading/locking rocket scientist. Most mere mortal developers are better to use message queues. Even the Java world has mostly given up locks in favor of java.util.concurrent which was implemented by serious experts to handle all of the corner cases that you would not think of. Third, using an external Message Queuing system like RabbitMQ gives you other benefits. And fourth, writing distributed apps glued together by message queues helps you avoid the dreaded Big Ball of Mud.

At this stage in Python's evolution, I view the GIL removal as a computer science project that some people will implement again, and again, just to learn or to exercise their chops. Great idea! Just don't demand that the entire community of Python developers goes down your road.

If CPython never gets rid of the GIL that suits me just fine. GIL free programming can be done on other implementations of Python like Jython and IronPython. As far as PyPy is concerned, as long as it does not disrupt the use of PyPy as a means of speeding up a CPython app from time to time, then have fun.

<comment><USER41> Coming from mobile and desktop programming, most use cases I've seen for threads revolve around doing something in the background to keep user interfaces responsive. That use case already has a threadsafe queue. The UI queue.

When your thread finishes or is ready to signal progress, you queue the event to the UI thread and forget about it.

Now I've been following this pattern for a long time and have no a experience dealing with GIL How this removal of GIL going to effect this use case if at all?

</comment></comment><comment><USER42> Just reading this post makes me think that it could do with a bit more "marketing" speak. I love python. I use it day to day and realize there is a GIL.

But give me some business reasons as to why removing the GIL is critical. Will is save me a ton of money? Will my stack magically just run faster?

I wonder if Google has already done so since they would benefit quite a bit from a GIL-less python.

</comment><comment><USER43> > We have some money left in the donation pot for STM which we are not using; > according to the rules, we could declare the STM attempt failed and channel > that money towards the present GIL removal proposal.

I didn't donate to that pot but that does seem like a judicious and reasonable step to take given the assessment of STM.

</comment><comment><USER44> I just made a quick test: CPython-3.6.1 vs. Jython-2.7.0 (May 2015)

I ran Larry Hastings' Gilectomy testprogram x.py: fib(40) on 8 threads HW: MacBook Pro, 2015, 8 (4+4) cores, 1Gb RAM

Jython ran the program 8 times faster, utilising all 8 cores >95%. Python ran on 1-2 cores less than 60% utilisation. (Pretty sure Jython will run 16 times faster on 16 cores)

It's 2017, why this is acceptable to GvR and the Python community is beyond me.

Jython: real 1m4.959s user 7m38.521s sys 0m2.396s

Python: real 8m19.035s user 8m16.508s sys 0m11.424s

<comment><USER45> Could you post the code for that fib program I couldn't find it anywhere.

<comment><USER44> It is in the Gilectomy branch of Larry Hastings github project. ([https://github.com/larryhastings/gilectomy](https://github.com/larryhastings/gilectomy))

I also pasted the fib test to pastebin: [https://pastebin.com/Ryyb2K7V](https://pastebin.com/Ryyb2K7V)

<comment><USER45> Ah so just the naive recursive fibonacci on 8 threads with no data sharing between them.

Interestingly doing the same on Cpython using the multiprocessing module was ~2x slower than jython/threads. More interestingly pypy with multiprocessing was ~5x faster than jython/threads.

      $ time jython fib.py 40
      real	1m11.247s
      user	6m14.130s
      sys	0m3.012s

      $ time python fib.py 40
      real	2m4.067s
      user	11m46.103s
      sys	0m2.352s

      $ time pypy fib.py 40
      real	0m21.040s
      user	1m51.461s
      sys	0m1.892s

</comment></comment></comment></comment><comment><USER46> Super glad they're going to try using mutexes and not that STM approach which was looking to be immensely complicated. Was not looking forward to the kinds of interpreter bugs that was going to produce.

</comment><comment><USER47> I can't believe it's 2017 and the official pypy updates come from blogspot; I thought this was a plea from a community member.

Anyway, really good on them to finally move on killing the GIL. It's been a long-time issue - the type that only gets worse the longer you ignore it. That said, I think today Python and GIL are synonymous and the entire Python ecosystem has almost evolved around the GIL. While I'm sure there are applications that would benefit from its removal, I think in the whole, the ecosystem will not change much because of this.

</comment><comment><USER48> Perhaps a little unrelated, I used the rpyc package to get Jython and CPython working together. In the end I was able to use Java libraries from CPython pretty much seamlessly.

<comment><USER49> You mean you used RPyC at both ends, on the CPython side and on the Jython side. Cool idea. I knew about RPyC but had not thought of using it in this way. And getting access to Java libraries by doing this, can be very useful, I can see.

</comment></comment><comment><USER50> Great, and I want be a billionaire with washboard abs. The main problem is none of the Python code that currently exists is thread safe so you might as well start again from scratch. Python is a needlessly complicated language with two important things; Numpy and TensorFlow. These use Python as a scripting language for C. Just move to Go, Scala or Elixir/Erlang if you want to avoid the GIL (or write anything parallel). You can thank me later!

</comment><comment><USER51> Do people here use pypy in production? What are the benefits?

<comment><USER52> Sure. Free 2-5x speedup. pypy + pypy's pip generally works as a transparent drop-in replacement to python + python's pip, so it's free speed.

It doesn't (or didn't) work when you need to rely on an extension that uses Python's C API. I haven't followed the scene in awhile so maybe that's changed. pypy's pip has so many libraries that I hardly notice, so maybe they solved that.

Unfortunately python is fundamentally slower than lua or JS, possibly due to the object model. Python traps all method calls, but even integer addition, comparisons, and so on are treated as metamethods. That's the case for Lua too, but e.g. it's absurdly easy to make a Python object have a custom length, whereas Lua didn't have a __len__ metamethod until after 5.1. I'm not sure it even works on LuaJIT either. Probably in the newer versions.

<comment><USER53> I can't tell what you mean by the last paragraph there, but oftentimes PyPy's speedups come exactly from inlining stuff like what you refer to there -- Python's not fundamentally slower, it's those kinds of stuff that you can speed up.

(And yeah the CPython API is still a pain point if you've got a library that uses it, although some stuff will still work using PyPy's emulation layer. It'd be great if people stopped using it though.)

<comment><USER52> For example, Python makes it fairly easy to trap a call to a missing method, both via __getattr__ and __missing__. In JS the only way you can do that is via Proxy objects, and even those have limits.

You can't always inline the arithmetic ops effectively. You can recompile the method each time it's called with different types, but that's why the warmup time is an issue. This wouldn't be a problem if Python didn't make it so trivial to overload arithmetic. JS doesn't.

<comment><USER53> Ah! Yes, agreed, Python does certainly make it too easy to do things that cannot reasonably be sped up.

<comment><USER52> Twist: Lua makes it trivial to overload arithmetic using metatables, but LuaJIT seems to have solved that. If there is any warmup time, it's hard to tell. Mike Pall is a JIT god, and I wish we had more insight into everything that went into producing one of the best JIT's of all time.

I'd love a comment/post that highlights the differences between JS and Lua _as the reason why_ LuaJIT was able be so effective. There must be differences that make Lua possible to speed up so much. There are easy ones to think of, but the details matter a lot.

EDIT: I found some discussion at [https://news.ycombinator.com/item?id=1188246](https://news.ycombinator.com/item?id=1188246) but it left me wanting more.

Related:

[https://stackoverflow.com/questions/4911762/why-is-luajit- so...](https://stackoverflow.com/questions/4911762/why-is-luajit-so-good)

[http://article.gmane.org/gmane.comp.lang.lua.general/58908](http://article.gmane.org/gmane.comp.lang.lua.general/58908)

[http://lua-users.org/lists/lua-l/2010-03/msg00305.html](http://lua- users.org/lists/lua-l/2010-03/msg00305.html)

<comment><USER52> More: [https://www.reddit.com/r/programming/comments/1r2s82/lua_fun...](https://www.reddit.com/r/programming/comments/1r2s82/lua_fun_is_a_highperformance_functional/)

</comment></comment></comment></comment></comment></comment></comment><comment> cool-RR Python's GIL issue is like the Israeli-Palestinian conflict.

1\. People like to talk about it a lot, complain about it and say their opinion of what should be done with it.

2\. It's not likely to be resolved for years to come.

3\. In the end, the problem has very little effect on people's lives, much much less than the amount of hype around the issue.

</comment><comment><USER38> Hi, blog post author here. Let me put an offer here:

If you want to ask a question that warrants a response (as opposed to promoting your own effort, which is valid but does not warrant a response), please mail me, the mail is public and I'll put the responses publically on either my blog or pypy blog.

</comment><comment><USER54> >fully working PyPy interpreter with no GIL as a release, possibly separate from the default PyPy release

I have concerns that if such functionality will not be in the main release enabled by default(and consequently don't get as much testing), it will just bitrot and in the end, will be removed.

<comment><USER55> They're asking for funding to spend on a risk-free attempt at GIL removal (risk-free since it won't bone PyPy mainline), if the attempt meaningfully succeeded I'd imagine their next step would be making it the default.

A fully functional PyPy that could do heavy math in multiple threads would be an amazing tool in the box, but there are plenty of risks to that (penalizing single threaded performance, for example). So this strategy makes plenty of sense to me.

They can't just do it on mainline from the outset because there are huge obstacles to overcome.. for example, that ancient foe, CPython extension interface compatibility, which assumes a single global lock covering all mutable extension data. I don't think there will ever be a way around maintaining the GIL for that, even if pure Python code can freewheel it otherwise

</comment></comment><comment><USER56> It's not going to happen, you not only have to fix all the legacy code, but also fix the developers.

</comment><comment><USER57> I liked "STM" as a big-idea approach, but can see how going the traditional way may bear fruit more quickly.

Could the experience gained this way (and by other projects such as the gilectemy) help with a future STM attempt ?

I wonder if we need better hardware for STM to work well too.

</comment><comment><USER58> Just curious: if they solve it in Python, would it be possible to solve it in Ruby too ?

<comment><USER59> If I understand correctly the issue in Ruby is the existing C extensions that have been written to assume the lock exists...

<comment><USER60> It's the same issue with Python. AFAIK there are a number of Python libraries that are not thread-safe, and the GIL prevents them from being an issue.

<comment><USER61> I thought the GIL was not held during execution of foreign code in python (at least that was one point given for why the GIL wasn't a big deal in practice).

<comment><USER62> No, it must be explicitly released. The GIL must be held to invoke almost all Python runtimes (main exceptions: acquiring the GIL, low-level allocator).

</comment></comment></comment></comment></comment><comment> unkown-unknowns > If we can get a $100k contract, we will deliver a fully working PyPy > interpreter with no GIL as a release, possibly separate from the default > PyPy release.

If done as a separate release, will that version be maintained in the future?

</comment><comment><USER63> Think about a recursive function whose implementation is changed while it is running. The replacement might have an entirely different algorithm. Which version finishes the stack call?

<comment><USER64> The version that was originally activated. I think that's the case in every single parallel implementation of a programming language ever. I can't imagine it working any other way.

When you redefine a method in any language I'm aware of you just change which method the name points to. You don't modify the original method.

<comment><USER63> So a function:

        def fun(*args):
            if not args:
                return 0
            return fun(*(args[1:]))

would be call-by-address after the first invocation? It could be lookup-by- name by way of code.

<comment><USER64> The naive implementation, and the semantic model, is always lookup-by-name on every invocation.

In practice we apply speculative optimisations including inline caching and guard removal with remote dynamic deoptimisation via safe points to make it a direct call instead.

</comment></comment></comment></comment><comment><USER65> Would this work with cpython extensions that were ported to PyPy?

<comment><USER38> They would run under GIL (I can't see CPython C API being thread-friendly unless gilectomy succeeds)

<comment><USER65> Ah, ok. So this approach doesn't completely remove the GIL, but removes it as a barrier for pure python code running in PyPy?

Or does it break the current support for porting cpython extensions?

<comment><USER38> It removes it for pure python code. The C extensions run under the lock (which is unfair to call _interpreter_ any more)

</comment></comment></comment></comment><comment><USER66> Sub-interpreter looks like an interesting idea, I don't mind limited to few primitive immutable objects shared between threads as long as it's shared.

</comment><comment><USER67> Does it work also in ARM architecture ("weak" memory model), or just in x86/x86-64-like ("strong" memory model)?

</comment><comment><USER68> Will this work with CFFI?

</comment><comment><USER69> This is a PERFECT usecase for Kickstarter. It makes me sad that this is a blog post that made it number 1 on HN with vast readership with open pursestrings.. yet there is not a campaign fundraising link.

Use Kickstarter or Plasso to sell a pypy pro license - its so much easier for companies to pay invoices than to donate.

If nothing else, I would pay for an official conda pypy package which works seamlessly with pandas and blas.

<comment><USER70> > yet there is not a campaign fundraising link.

Did you read the article? They said in the article they aren't asking for individual donations at the moment:

>> we would like to judge the interest of the community and the commercial partners to make it happen (we are not looking for individual donations at this point)

Plus I'm sure they will consider using Kickstarter when the time comes.

<comment><USER71> Cash is one objective way to discern interest.

</comment></comment></comment><comment><USER72> Removing it is the easy part.</comment>


gRPC-Go Engineering Practices - <AUTHOR> https://grpc.io/2018/01/22/grpc-go-engineering-practices.html <comment><USER2> Its a good time to ask, is gRPC any good? I'd love to standardize on stable middleware layer that handles multiple versions of clients and servers well. Rest with json really seems to work great for most things already.

What is the advantage of gRPC - just more efficient?

<comment><USER3> Afraid I'm going to be contrary and old-fashioned and say I prefer JSON.

Its never been problematic adding or extending JSON endpoints and its never been a problem using basic gzip compression on the fly either.

And JSON endpoints are a damn sight easier to debug and wireshark and all the rest.

I've spent a lot of time writing fast JSON serialization for various languages including Java etc; its staggering how inefficient most libraries are. But that's not really JSON's fault.

<comment><USER4> There are definitely cases for both. JSON is definitely easier to consume as a human (so for debugging).

gRPC was born out of Google's Stubby rpc system[0], which is used heavily for communicating between different jobs. If you are going to stand up a lot of different services that are going to talk, it provides a lot of advantages that you don't get with JSON. For large companies that use multiple program languages, this is really nice, as proto3 and GRPC have code generators for a slew of different languages.

There are a lot of other niceties that are useful in gRPC that you don't get with JSON (like streaming data).

[0] [https://grpc.io/blog/principles](https://grpc.io/blog/principles)

<comment><USER5> Also a protobuf may be stored directly in some of the databases google uses internally, then you have lots of tools around them (diffing with map-reduces done on them, etc.).

</comment></comment></comment></comment><comment><USER6> Slightly off-topic, but related.. I have read that a common practice for managing proto files (or any schema definitions really) is to put them in a separate repo/package to share. It seems pretty straightforward in my head and provides several advantages. However I still ask about any trade-offs when doing this in practice?

<comment><USER7> It has similar challenges than a monorepo.

First challenge is, that you have to keep some kind of reference on what proto are you using within your project. What Google (and some others) do, is that they a) put all the proto files in a separate repo [0] and then generate them for each language separately (python [1], ...). This way you can use whichever proto file you need within your project, however you have to load more libraries than you need to. To be honest, it only makes slight difference when deploying, so not too bad.

The second challenge is, that you have to generate the result files every time you make some kind of change. If you have a lot of proto files, then it may take some time to generate them and there are very little tools available to help you. Google open-sourced Artman [2] although it's more focused on APIs than managing shared protos.

The massive advantage is that, because proto files are self-explanatory and if you put enough information in them can function as direct documentation of your API's interface, you don't need to fish out the requirements in the project or in the documentation but rather just directly read the proto file itself. But this does depend on the developers, to make it as consistent as possible, which is not always the case [3].

[0] [https://github.com/googleapis/googleapis](https://github.com/googleapis/googleapis)

[1] [https://pypi.python.org/pypi/googleapis-common- protos](https://pypi.python.org/pypi/googleapis-common-protos)

[2] [https://github.com/googleapis/artman](https://github.com/googleapis/artman)

[3] [https://news.ycombinator.com/item?id=16166153](https://news.ycombinator.com/item?id=16166153)

<comment><USER6> Thanks for the response! These are very good references. I like the breakdown of message and/or service definitions into separate files so it is easy to browse through the repository. I am assuming the version bump (v1 vs. v2) occurs when you introducing breaking changes to the API? I am aware that protobuf does a good job at allowing forwards and backwards compatibility at the message level..

<comment><USER7> Correct on the versioning. We internally treat protos as an immutable descriptions of our API interfaces. That means whenever we need to change anything (outside of bugs), be it adding a new field, or changing the order, renaming fields, changing types, ... we start a new version.

We also use a lot of inheritance of non-default types (timestamps, errors, ...) so it's important to make sure that we don't break anything for others.

<comment><USER6> Interesting so adding new fields even.. I suppose there is quite a bit of planning and iterating on the interface before making it public to reduce churn? I understand doing that for backwards incompatible changes.

<comment><USER7> We add new fields (or redesign existing protos) very sporadically. We also don't have fully automated process to deal with a change in a proto within all projects so by using new versions we can signal to the developers that they should eventually adopt it. We are actively using 5 languages that have to deal with the changes, so we found it easier overall to do it this way.

</comment></comment></comment></comment></comment></comment><comment><USER8> Great to hear, I'm pretty bullish on the framework and have been using it happily in Go for a while

</comment><comment><USER3> There has been a flurry of gRPC posts on HN recently - it must be the new xml soap rest NoSQL fad!

NoSQL is an interesting parallel - Google publish map reduce and bigtable and amazon publish some influential papers and suddenly everyone is using NoSQL in order to be "web scale". Then it turns out that Google themselves were doing sql web scale and spanner and all that.

There's a risk that gRPC is the same? In chat yesterday ex-googlers said that Google was increasingly moving over to flatbuffer...

Personally I have an aversion for tools with generators. Harks back to the damage CORBA did me I guess... I also have a preference for plaintext eg JSON - so much easier to debug.

Oh well. Guess we're in the fashion business ... ;)

</comment><comment><USER9> Is this gRPC the same thing as golang net/rpc referred to here: [https://news.ycombinator.com/item?id=16170116](https://news.ycombinator.com/item?id=16170116)? I don't think so but I've never used either one.

>seniorsassycat: _I don 't understand why AWS released Go support instead of binary support and I don't understand why they chose to rely on go's net/rpc [...] which encodes objects using Go's special [gobs] binary format_

<comment><USER10> net/RPC is a rpc implementation in the go standard library, which uses gob for serialization.

gRPC is a protocol and set of libraries for cross-language rpc based on protobuffs. Also doing a lot of codegen for you, like generating clients.

</comment></comment><comment><USER11> Has anyone found a good resource on using gRPC directly in a JS client? I've looked at using gRPC. My current challenge is that I want to support a website/webgateway on one side and mobile gateway on the other. If I use Swift or Java on the mobile side, it's easy. If I use Ionic Framework, I'm in the same spot as with the web gateway; probably better off with HTTP + RPC.

<comment><USER4> Does this work? [https://github.com/grpc-ecosystem/grpc- gateway](https://github.com/grpc-ecosystem/grpc-gateway)

Seems to generate a REST proxy server side.

<comment><USER11> I haven't tried it, but that goes against the grain of gRPC. I understand it as an integration point between legacy REST and gRPC. I don't know, and from what I've read, wouldn't use it for greenfield development.

The only issue I have with full HTTPS + JSON is CRIME. I'm not sure how that works.

</comment></comment></comment><comment><USER12> Does anyone have experience with both gRPC and Thrift? I'd be curious to know how they compare.

<comment><USER13> Thrift used to support more languages (this has changed). gRPC was more performant (take w/ grain of salt, this is word of mouth) for a while -- unclear if it's changed or if the difference was ever that significant except at very high scale.

I think they're pretty similar and you can't lose either way. Facebook's support of Thrift and Google's of gRPC make both decent options.

One thing I will say about gRPC is that it plays nice with Google's build system (Bazel) and some Google APIs now have first-class gRPC support. If you choose thrift in your stack you'll have to call APIs using JSON or support gRPC anyway if you want to use them for those API calls...so gRPC might be an attractive choice. Furthermore gRPC's go interop is also excellent if you happen to be a fan of golang.

<comment><USER14> When I looked into Thrift's generated code for Objective-C when we started working on gRPC (so 2014), RPCs were all blocking, which made for a non- idiomatic API. I haven't looked if they've added an asynchronous version since.

</comment></comment></comment><comment><USER15> Is it possible to add a middleware which does gRPC <-> JSON?

<comment><AUTHOR> Check out grpc-gateway[1]

[1]: [https://github.com/grpc-ecosystem/grpc- gateway/](https://github.com/grpc-ecosystem/grpc-gateway/)</comment></comment>


Secret court judge rejects US gov't request for longer metadata retention - <AUTHOR> http://arstechnica.com/tech-policy/2014/03/secret-court-judge-rejects-us-govt-request-for-longer-metadata-retention/

<comment><USER2> _in a win for civil liberties advocates,_

Stop me if I'm wrong, but nobody advocates against the government lawyers in FISC arguments, right? So, how can this be a win? I know that's a technical point, but still...

If a notorious rubber stamp says the arguments are "simple unpersuasive", how bad are the gov's arguments?

This also shows that even a hint of publicity works wonders as far as getting some justice out of a secret court. One has to question what other massive abuses are going on under the cover of "national defense" or "compartmentalizatoin"/</comment>