Employing QUIC Protocol to Optimize Uber's App Performance - <AUTHOR> https://eng.uber.com/employing-quic-protocol/ <comment><USER2> This YouTube video does a great job illustrating how well HTTP/2 works in practice.

[https://www.youtube.com/watch?v=QCEid2WCszM](https://www.youtube.com/watch?v=QCEid2WCszM)

A lesser known *ownside to HTTP/2 over TCP solution was actually caused by one of the improvements - a single reusable (multiplexed) connection - that could end up stalled or blocked due to network issues. This behavior could go unnoticed over the legacy HTTP/1.1 connections due to browsers opening a hugh number of connections (~20) to a host, so when one would fail it wouldn't block everything.

<comment><USER3> It's an unrealistic test as pages aren't really made up of tiles of images

Generally images are the lowest priority download, so ensuring higher priority items get downloaded first is important and not all H2 implementations do it well

[https://ishttp2fastyet.com](https://ishttp2fastyet.com)

<comment><USER4> Its only unrealistic because so much tooling was built to avoid sending multiple files. JS tools for bundling every js file in to one, sprite sheets, using multiple domain names to get more concurrent connections. With HTTP2 we could dump so much of this.

<comment><USER3> To me it's unrealistic in the sense that it's an artificial test

Images are the lowest priority resource on the page and apart from the visual appearance aspects there are no dependencies on which order they're fetched in.

Most other resources on a page have greater side effects and and dependencies e.g. sync JS blocking the HTML parser, sync and deferred JS need to be executed in order etc.

You can saturate a last mile connection with image downloads in a way you man not be able to with other resources due to effect of the browser processing those resources.

</comment></comment></comment></comment><comment><USER5> What a great case study. Successfully shifting 80% of mobile traffic to QUIC for a 50% reduction in latency is amazing. QUIC and the ongoing work with multipath TCP/QUIC will be huge QoL improvements for mobile networking.

<comment><USER6> The other awesome thing about QUIC is that it encrypts almost everything including header information, making middlebox traffic shaping worthless and demoting middleboxes in general.

<comment><USER7> It also makes hardware offloads like TSO and LRO impossible, and increases cost-per-byte served by a factor of 4 or more. So if you have infinite CPU to throw at QUIC and/or low bandwidth or connection targets, its great. If you are concerned at all about server-side efficiency, its terrible.

FWIW, I work on the Netflix CDN, and specialize in server-side efficiency; we have had 100G flash CDN nodes for years serving at 90G+ in production. None of that would be possible with QUIC as it stands. I suspect our max B/W on these machines would drop from ~95Gb/s to 20Gb/s or less if we were to switch to QUIC.

<comment><USER8> Does the protocol actually make it impossible, or is it just not implemented by current OS/hardware?

<comment><USER9> Definitely just that the HW hasn't caught up.

[https://www.netdevconf.org/0x13/session.html?talk-quic- offlo...](https://www.netdevconf.org/0x13/session.html?talk-quic-offload)

</comment></comment></comment></comment></comment><comment><USER10> Experiment 1

 _While we used the NGINX reverse proxy to terminate TCP, it was challenging to find an openly available reverse proxy for QUIC. We built a QUIC reverse proxy in-house using the core QUIC stack from Chromium and contributed the proxy back to Chromium as open source._

Experiment 2

 _Once Google made QUIC available within Google Cloud Load Balancing, we repeated the same experiment setup with one modification: instead of using NGINX, we used the Google Cloud load balancers to terminate the TCP and QUIC connections...

Since the Google Cloud load balancers terminate the TCP connection closer to users and are well-tuned for performance, the resulting lower RTTs significantly improved the TCP performance._

</comment><comment><USER11> I recently moved and got internet with Spectrum. A 200/10 service yet my upload speeds were rarely above 5mbit. This was a business account with some web and dev servers behind it. I didn't even try to call customer service...

With a little more testing using UDP, I could see I was getting very spotty packetloss (<0.5%). I'd never tried changing the TCP algo before but I knew random packetloss is normally interpreted as congestion and hence causes a speed backoff.

I tried all of the ones available at the time but the one that stood out not only in performance but also simplicity was TCP-Illinois[0]. The stats provided by `ss -i` also seemed the most accurate with TCP-Illinois. I force enable it on every machine I come across now.

0:[https://en.wikipedia.org/wiki/TCP- Illinois](https://en.wikipedia.org/wiki/TCP-Illinois)

<comment><USER12> I highly recommend BBR congestion control if your router supports it.

<comment><USER13> TCP congestion control is end-to-end. Routers don't need any support for it.

<comment><USER14> This previous discussion will probably help here. [https://news.ycombinator.com/item?id=14298576](https://news.ycombinator.com/item?id=14298576)

</comment></comment></comment></comment><comment><USER15> Tcp was build for the internet long ago, even though there are changes added, the architecture of the protocol make it hard to do anything drastic. With UDP because it is so simple, you can basically create a new protocol on top, inside the payload and emulate TCP if you wanted to

</comment><comment><USER16> I wish the mandated minimum MTUs of IP were just a bit bigger. Ubers traffic must be so transactional, they could really just use individual UDP packets for most messaging.

<comment><USER17> ipv4 - 576 bytes

ipv6 - 1280 bytes

<comment><USER18> That should be sufficient for uber, I think.

<comment><USER16> >>> print len("Dear Uber, I need a ride from 26th and 6th Street, Brooklyn to 999 Broadway. I am on the south side of the road. It's 6:99PM UTC, I have 55% battery, my public key is b82c9238e847b. I like Jazz. My phone is a Pixel 3, I am using app version 4.22.4, but you probably already knew that. I am running out of things to add, so here is some personally identifiable information... and we still have plenty of bytes left")

Edit: formatting

<comment><USER19>
          File "<stdin>", line 1
            print len("Dear Uber, I need a ride from 26th and 6th Street, Brooklyn to 999 Broadway. I am on the south side of the road. It's 6:99PM UTC, I have 55% battery, my public key is b82c9238e847b. I like Jazz. My phone is a Pixel 3, I am using app version 4.22.4, but you probably already knew that. I am running out of things to add, so here is some personally identifiable information... and we still have plenty of bytes left")
                    ^
        SyntaxError: invalid syntax

</comment></comment></comment></comment></comment><comment><USER20> So is this essentially HTTP/3?

<comment><USER21> Yes. [https://tools.ietf.org/html/draft-ietf-quic- http-20](https://tools.ietf.org/html/draft-ietf-quic-http-20)

</comment></comment><comment><USER22> Isn't TLP[0] supposed to fix the largest cause (tail losses) of this issue? It should result in retransmits far sooner than the 30 seconds they mention.

> Recently developed algorithms, such as BBR, model the network more > accurately and optimize for latency. QUIC lets us enable BBR and update the > algorithm as it evolves.

Again this is available for TCP in recent linux kernels[1]. And it's sender- side, so it should be unaffacted by ancient android devices.

Are they using ancient linux kernels on their load balancers? Or are the sysctl knobs for these features turned off in some distros?

[0] [https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=6ba8a3b19e764b6a65e4030ab0999be50c291e6c) [1] [https://kernelnewbies.org/Linux_4.9#BBR_TCP_congestion_contr...](https://kernelnewbies.org/Linux_4.9#BBR_TCP_congestion_control_algorithm)

<comment><USER23> BBR in kernel works for TCP because it's used for TCP congestion control. QUIC is built on top of UDP and there is no congestion control used for UDP, you implement it in your application/protocol built on top of UDP. From what I remember QUIC implementation supports BBR and Cubic as congestion control mechanisms.

<comment><USER24> Correct. QUIC also implements TLP. Main advantage is deployability since this is a user-space solution.

</comment></comment></comment><comment><USER25> I thought DDOS prevention was difficult with udp, compared to TCP. Is it not the case anymore. Does cloudflare provide DDOS prevention for QUIC/UDP.

<comment><USER14> UDP DDOSes are hard to prevent against because they're volumetric in nature. Generally, they rely on UDP's statelessness, poorly configured networks, and various applications that respond with significantly more data than requested over UDP. These types of DDOSes are useful against any service that's running on TCP or UDP, and Cloudflare has protected against them all along.

</comment></comment><comment><USER26> I'm surprised the "alternatives considered" section doesn't have a "write something custom for core functionality using UDP". I would be curious to read why the decided not to go that way, given their scale and the potential gains from not using a general-purpose protocol.

(Something like, make the entire standard journey from opening to the app to requesting a car over something custom, and then leave the rest of the app using TCP)

<comment><USER27> I'm assuming their thought process went like this: "we want to optimize a workflow which today goes over HTTP RPC/REST for mobile networks where the retry behavior of TCP at the low level is suboptimal. QUIC already exists, and we don't have to figure out our own solutions to firewalls, security, HTTP semantics. Also our webapps use the same or similar API endpoints, so the less effort spent writing new routing, monitoring, etc., the better. Oh, it works really well. Awesome, avoided 6-24 months debugging a custom protocol over mobile networks around the world that we don't control."

</comment></comment><comment><USER28> This kind of latency improvement only matters if they are planning to do auto- pilot from the cloud? (that would be crazy, especially if they don't have a fallback)

<comment><USER29> Does it necessarily imply that, though? applications that are snappy and responsive tend to improve users' impression of the app and probably (it is/was true for browsers when I worked in this realm, I have no reason to doubt that it's true for mobile today) improve user engagement.

<comment><USER30> I think the prevailing wisdom is that engagement suffers more from latency on mobile.</comment></comment></comment>


I support Gus - <AUTHOR> http://supportgus.dk/

<comment><USER2> Whenever I read about a campaign to stop someone from being deported, I am almost always sympathetic and supportive. But I also can't help but think about the people who are no less deserving of support, but don't have a community with the means/wherewithal to launch such a campaign behind them. Yes, people should work hard to stop Gus from being deported. But if the campaign stops once Gus' situation is resolved the campaign will be a minor victory at best. Gus' situation is a symptom of a disease. It is the disease itself which must be cured

<comment><USER3> Very true, but just because you can't help everyone doesn't mean you shouldn't help someone.

</comment></comment><comment><USER4> I'm Danish, and unfortunately this story doesn't surprise me. My brother's Australian girlfriend was unable to get a residence here (He's an engineer, she's a lawyer), one of my good friends' girlfriend from Brazil couldn't get a residence here even though they have a five year old son that's born and raised in Denmark. All of them are hardworking people that can take care of themselves. The list goes on.

The problem is that around ten years ago the current conservative minority government was formed, and to gain a majority they have relied on the right- wing Danish people party whose primary, and maybe only agenda is stopping immigration. This has lead to a political reality where the Danish People party has voted yes to whatever legislation from the government in return for further restraints and barriers for immigrants. They've used their position extremely well, and have used their platform to alienate immigrants and make them all out as devils in the minds of the people.

Ten years of this has unfortunately marked the country. The tone in the media is increasingly harsh towards immigrants, and other parties have started adopting the stance of the Danish People party since it's pretty successful - after all the Danish People party has had a decade to convince Danes that immigrants are only here to steal our jobs and use our social services.

As a result we've become a country of people that despise immigrants.

It's a very sad state of affairs, and this website is only one of many examples. I feel terrible about what my country has become.

<comment><USER5> It feels like the same is true for the UK and the USA – perhaps the whole western whole? That particular political situation doesn't sound ideal though.

</comment></comment><comment><USER6> I really don't understand Denmark when it comes to migration issues. They make it so hard to move there for non-EU citizens that it is ridiculous.

As an example, I know of more than one married couple with one Dane and one non-eu citizen that have been forced to live in Sweden instead of in Denmark due to troubles getting residency permits. Not letting a legitimate spouse get at least residency is just crazy.

<comment><USER7> As a Dane I can only say I don't understand it either. But I can give you a couple of small hints as to why it (unfortunately) is like it is.

1\. Denmark has a very beneficial social welfare system.

If you break your leg in Denmark visiting, we will pick up the bill through our taxes (there is talks about changing that now)

If you get approved to stay here you gain access to more or less all social welfare.

Some people feel that this is being misused by immigrants. When they normally talk about immigrants they talk about arabs, africans etc. And there is some truth to that.

If you marry someone not from the EU they will gain access to all social welfare.

Because the nineties where ruled by a government that "just took people in" it created a backlash as problems started arising with ghettoes. This among other things lead to a (european style conservative) government backed by Danish Peoples Party a political party critical towards immigrants.

Since they secure the parliamentary power they have been able to get a lot of cases through.

One of them being the 24 year rule which basically is meant to hinder forced marriage between muslims living in Denmark and their spouses living in a country outside of EU.

This has been criticized by both left and right side of the political spectrum and is why some people move to sweden because they are much more lax about these things.

Personally I am a proponent of an open borders closed boxes policy and is voting for the only true liberal (semi american style) party that exist right now (form an ideological point of view) and I am sad to see every time people like Gus get's en trouble because of the splash damage that some of these laws result in.

Good thing is that Denmark is a small country and there is an election year. I am pretty sure that this will be taken up by the newspapers in Denmark and hopefully make it's way into the political debates on television. It's a perfect case for that.

I am also pretty sure Gus will be able to stay. The current government simply don't want to run the risk of angering their political base.

<comment><USER8> _Some people feel that this is being misused by immigrants. When they normally talk about immigrants they talk about arabs, africans etc. And there is some truth to that. If you marry someone not from the EU they will gain access to all social welfare._

True. I am from another welfare state dealing with the same problems and discussions (The Netherlands). But it needs to be said that the system was grossly misused by native Dutch as well. Laws have been tightened since the beginning of this century, but some groups continue to be resourceful in exploiting the system.

Also interesting is that the anti-immigrant party is most popular in a low- immigrant, economically weak region (Limburg). It's highly paradoxical that people in regions that rely a lot on social welfare accuse others of bankrupting the state (immigrants, The European Union, and state officials are popular targets).

<comment><USER7> Same thing here. People in Copenhagen areas where most immigrants live are pretty liberal when it comes to immigration. Where as Jutland and other places are more conservative.

But then again they are the ones fighting with unemployment and declining population so someone need to get the blame I guess.

</comment></comment></comment></comment><comment><USER9> An informative comment by a Dane in this thread suggests that the Danish immigration system tries to prevent immigrants using/exploiting its great social welfare benefits while contributing little to the society or causing problems like forming ghettos.

A straightforward way to address this issue is to deploy a point system currently used by several countries, including Canada and Singapore.

I wonder why Denmark and the US, in particular, do not use this obvious sort of point-based system. There will likely be a massive 'brain draw' towards the US if this is implemented and specialized Startup Visa may not even be needed if the criteria/formulas are selected well.

In the Danish system, it looks like the criteria are not optimized towards its real goals and use non-optimal proxy criteria like EU citizenship to make decision on immigration. This creates many weird cases like this (it is obvious from his resume that Gus will contribute as much or more to the Danish society than an average Danish citizen, yet he cannot stay there) and many we did not hear of.

Note: It is true that the politicians needs to balance the issues of cultural assimilation with economic advantages gained from getting all these qualified immigrants (for example, the US might not want 2 million qualified Chinese and Indians coming in every year and form cliques within the country), but this requirement is easily satisfied by a good quota/bucket system.

<comment><USER10> _A straightforward way to address this issue is to deploy a point system currently used by several countries, including Canada and Singapore._

Plainly, to tolerate laws that explicitly discriminate against low-income and blue collar workers.

<comment><USER11> The Canadian list of "skilled workers" is a list of job types where they are current shortages. Some IT jobs were recently taken _off_ the list, and it contains plenty of blue-collar type jobs:

[http://www.cic.gc.ca/english/immigrate/skilled/apply-who- ins...](http://www.cic.gc.ca/english/immigrate/skilled/apply-who- instructions.asp#list)

You spent 1 year in the past 10 as a fry cook or installing flooring? Great! Canada needs your skills!

The Australian list contains plenty of blue-collar jobs too:

[http://www.workingin-australia.com/visa/skill- shortages/skil...](http://www.workingin-australia.com/visa/skill- shortages/skilled-occupations-list.aspx)

<comment><USER10> I don't claim that they _don't admit_ low-income workers, but that their laws discriminate in preference of high-income workers (that is, highly educated professionals). See their points system [1]. (It also discriminates on the basis of age, which would be a Human Rights violation if it was a private employer doing it. [2])

[1] <http://www.workpermit.com/canada/points_calculator.htm>

[2] [http://www.hrsdc.gc.ca/eng/lp/spila/wlb/caowc/10chapter_5.sh...](http://www.hrsdc.gc.ca/eng/lp/spila/wlb/caowc/10chapter_5.shtml)

<comment><USER11> Some random other country doesn't owe you a thing and so they get to be selective about who they let in. In Canada you have to show you won't place "excessive demand" on their services, and it's defined as:

"Requirements for medical or social services of such a level as to threaten the sustainability of Canada's publicly funded healthcare system, or deny or delay access to Canadians that would prevent the admission of an applicant to Canada."

[http://www.cic.gc.ca/english/resources/publications/dmp- hand...](http://www.cic.gc.ca/english/resources/publications/dmp- handbook/section-1.asp)

Some random web link says that Excessive Demand was calculated to be $24,030 of health or social services over a five-year period. There's lots of stories of families with a mildly autistic child being denied residence in Canada, for example.

[http://www.canadianimmigrant.ca/health/healthinsurancecovera...](http://www.canadianimmigrant.ca/health/healthinsurancecoverage/article/2532)

It's not in the best interests of a country to let an older person in who wont pay much tax and then try to collect social security and use health services as they get older.

Australia, for example, has a specific visa to enable retirees to live in Australia for a period of time. Note, however, that you need a lot of assets, a substantial income and proof of health insurance for the duration of the stay.

<http://www.immi.gov.au/visitors/retirement/405/>

</comment></comment></comment></comment></comment><comment><USER12> As long as Denmark has a growing "Danish People's Party" (3rd largest at 13.8%) in a government supporting role, with the one-trick pony voter's appeal of immigration (thus they have to tighten and tighten it), I have 3 advices for anyone contemplating Denmark: 1) Do not move to Denmark or start a company in Denmark 2) Do not get an education in Denmark 3) Do not fall in love with a Dane (if you are a non-EU citizen)

Danish educational institutions and companies go to a great extent to convince their surroundings, that they are international & business oriented, but the matter of fact is that this is not supported by the parliament. It is as simple as that. So don't get lured by their marketing. Stay out.

</comment><comment><USER7> As predicted it is now front page on one of the biggest danish newspapers <http://www.b.dk/>

<comment><USER13> I guess I underestimated the Danish papers, didn't think they would pick up on this. Good for Gus!

</comment></comment><comment><USER14> It is very difficult for me to take a side without hearing views from all the sides for a story. Google was not much help.

Any links where I can see the Denmark Government's Point of View of his kicking (any relevant documents etc.)?

<comment><USER15> The thing you can probably research is inter-department communication in Government agencies in Denmark. If its anything like Sweden then Government departments behave in a mostly autonomous fashion where decisions are made based on the departments mandate and prerogatives, irregardless of what decisions have previously been made by other Government departments. It can lead to some strange situations.

</comment></comment><comment><USER16> Thanks for all your support, so far my case has been picked up by several media outlets but I am yet to get a response from any of the political representatives. However, the fat lady hasn't sung so I won't be dropping the ball just yet. My case, I believe, highlights some of the inefficiencies and absurdities that are in place, within some governmental organizations. I fully recognize that because of my involvement with entrepreneurship and my network I have managed to gain a great amount of support to my case. Saying that, there are many smarter and more brilliant people out there who's case will remain silent and they'll just disappear into the night. I hope, if one thing, my case can be an example of what others can do to show that a country shouldn't be known by it's policies and procedures but by the amazing people within it that won't lay down but are willing to voice their disapproval to cases like mine. Thanks again for your support!

</comment><comment><USER13> Agree with all the points about Denmark; as a Swede I'm happy that we at least are somewhat better at making it possible to immigrate.

I'm very curious as to what impact this campaign can have on the Danish authorities. I would guess none if it doesn't get picked up by the Danish (print) media and since Gus isn't a classical victim (meaning poor, uneducated, a child and so on) I don't see that happening. Unfortunately.

That said, it's nice to see someone putting down actual work in creating a web site for a friend instead of the usual "create a facebook group" or "retweet to show support" which to me feels like the most lazy way to show that you care.</comment>


Functors, Applicatives, and Monads in Pictures – In Clojure - <AUTHOR> http://fluokitten.uncomplicate.org/articles/functors_applicatives_monads_in_pictures.html?hn=1

<comment><USER2> This is a bit old, but interesting.</comment>


A Regular Expression Matcher in 30 lines of C - <AUTHOR> http://www.cs.princeton.edu/courses/archive/spr09/cos333/beautiful.html <comment><USER2> It's very nice pattern-matching code, but it isn't a regular expression matcher. The class of patterns it's able to match is much smaller than the class of all regular expressions -- not because of missing "sugar" (character classes, +, non-greedy wildcards, ...) but because it doesn't support groups or the | operator.

As a result, the language supported by this matcher has, e.g., no way to match the pattern (ab)* or the pattern a|b. It's much, much less powerful than an actual regular expression matcher would be, and much of what makes it possible to do it in 30 lines of C is that loss of power.

(I've written extremely similar code before: this level of functionality -- basically, glob or DOS wildcards -- is pretty useful. I'm not Rob Pike, and my code for similar functionality would probably be longer than his. But my code, or even Rob Pike's code, for even the simplest thing that could honestly be called a regular expression matcher, would be longer than this by a bigger factor.)

</comment><comment><USER3> I see the OP has discovered [[http://www.hnsearch.com/search#request/submissions&q=cod...](http://www.hnsearch.com/search#request/submissions&q=coderdude&start=0)] r/tinycode which was posted on HN few days back [<http://news.ycombinator.com/item?id=2717279>].

</comment><comment><USER4> I wrote a similar glob-style string matching, but it's far more than 30 lines (130 lines of C):

[https://github.com/antirez/redis/blob/unstable/src/util.c#L1...](https://github.com/antirez/redis/blob/unstable/src/util.c#L11)

I use it in Redis, but in general being "glob matching" more standard compared to a regexp subset people are more likely to know the syntax.

</comment><comment><USER5> That was a superb read. Given the author(s) (who I noticed only after reading), I suppose that's to be expected. I'd like to see a similarly elegant version that translates the RE to a DFA–if anyone has any suggestions, I'd be quite happy!

</comment><comment><USER6> This an excerpt of the excellent book <http://oreilly.com/catalog/9780596510046>

</comment><comment><USER7> Yes, matching regular expressions is easy, if you don't care about speed or features. Like, being able to match a literal dot or asterisk. Or char classes, or getting case insensitivity right (like the German ß that matches SS, so case folding needs not to preserve string length).

Still nice to see that it can be done in so few lines.

</comment><comment><USER8> Here's my attempt at implementing the same functionality in Haskell: <https://gist.github.com/1062472>

I tried to make it short, but still somewhat readable, so it's not completely "golfed".

</comment><comment><USER9> "In 1967, Ken [Thompson] applied for a patent ..."

Wow, he pioneered that, too...

</comment><comment><USER10> That code is dense.</comment>


Show HN: Share your location using Longitude.me - <AUTHOR> http://longitude.me

<comment><USER2> This is hilarious, i just made exactly this as a personal project very recently (only its super ugly, i did it in under 1 day)

[http://ff.mouseofdoom.com/](http://ff.mouseofdoom.com/)

I also uploaded the source to my git account if anyone wants to take a look: [https://github.com/ndroo/friend-finder](https://github.com/ndroo/friend- finder)

</comment><comment><USER3> It works good. Glympse is out there that is an advanced version of this, but I like this.

I would like it better if it ran in the background. I'm not sure on battery use either. Did you use something that saves battery or is it constantly sending my location? Also an iPhone app that worked in the background that was smoother would be awesome. If you want some help on dev or design I'd love to help!

<comment><AUTHOR> Yeah, it constantly sends your position while you look at the page. I also wish it worked in the background, but unfortunately I don't think it's possible. It stops updating your position when you exit the browser app, or if you lock the phone (at least on iOS). I might do an app (that goes against the same backend) in the future!

</comment></comment><comment><USER4> Really, really nice. Can't count the number of times I've wanted something exactly like this.

And beautful site and maps too.

Great job!

<comment><AUTHOR> Thank you! Happy you like it :)!

</comment></comment><comment><USER5> With more gif: [http://www.meattext.com/](http://www.meattext.com/)

</comment><comment><USER6> What a beautiful map - what service are you using for that?

<comment><USER7> Looks like a service called CloudMade: [http://cloudmade.com/products/map- tiles](http://cloudmade.com/products/map-tiles)

<comment><AUTHOR> Yep, map tiles are from CloudMade! Map lib is Leaflet. I also made a Leaflet plugin for the user markers, which is on github: [https://github.com/heyman/leaflet- usermarker](https://github.com/heyman/leaflet-usermarker)

</comment></comment></comment><comment><USER8> aka

rob-my-house.net

<comment><USER9> Realistically, burglars prefer to rob houses where people aren't.

My guess is that very few burglars sit around scanning all of the (52 x 52 x 52 x 52 x 52 x 52) possibilities of Longitude.me directories (assuming they just use case sensitive letters), and then when they find one where there's a person (near them) THEN drive to that location and rob them.

If they're that hard pressed to rob a real-live person they could just go door-to-door until they found someone.

I can't back this up with research, however.</comment></comment>


Google Maps – the $10k gotcha - <AUTHOR> http://www.47hats.com/?p=1329

<comment><USER2> Google isn't omniscient. You probably won't get noticed by them until you are large enough to pay their fee. I wouldn't be surprised to find out that Google never enforces this $10K rule, and that it's there just in case.

And if you do get noticed, and they decide to enforce the rule, nothing is stopping you from making that one feature free.

<comment><USER3> They're not omniscient, but I wouldn't count on that. It wouldn't be all that hard to collect a unique list of URL's and have an intern go through a bunch of them per day. The list could probably be augmented pretty easily with some analysis of the page content to detect fee related text, and they certainly have the technology to make that happen. They can narrow it down pretty quickly too based on what key your using to access the API.

The terms are pretty clear. Better to pay it if your using it and charging for it up front rather than try to get clever and get caught later and potentially have legal fees to pay on top of it IMHO.

<comment><USER2> if Google did this before, I'm sure people would have heard about it by now. Right now it seems like they are choosing to ignore this.

Noone really reads terms of service, so I'm pretty sure the $10K fee wouldn't stick. What would probably happen when you'd get caught, is you'd get an email from Google telling you that you've broken their ToS and that if you want to continue using their service you'd need to either a) start paying $10K/mo or b) make the service public

</comment></comment></comment><comment><USER4> Something clearly spelled out in the terms of use of the API can hardly be called a _gotcha_.

<comment><USER5> Agreed. There's no surprise to me here that Google would want a piece of the action if you are making money with their freely available services.

</comment></comment><comment><USER6> Couldn't you just make the map publicly available for both users and non- users? Your fee/subscription-base isn't for the map itself. The map simply serves as a manner to visualize data.

I have a feeling that the many, many real estate companies whom use Google maps aren't all paying for the API, despite it being a commercial venture.

<comment><USER7> Yeah, but they offer the mapping aspect for free (far as I know). If the mapping is behind a paywall, I think that's where the pricing starts.

<comment><USER6> That's why I was saying you should make it available to both users AND non- users. That way if Google inquires (which I truly doubt they ever would), there is no way they can say the map is restricted to subscription/fee.

</comment></comment></comment><comment><USER8> _shrug_

Seems pretty fair to me. Not especially friendly to startups, but definitely fair.

<comment><USER9> It's not about fairness. The question from the point of view of Google is whether their pricing is optimal. Are they missing out on a significant market who would pay less for a cheaper option but not $10k for the Premier service? Only they know that.

</comment></comment><comment><USER10> What about iphone apps? Say you make an iphone app that uses the google map tiles and api, but you want to charge $.99 for this app, are you supposed to pay the $10k license fee??

<comment><USER11> Great question. IANAL, but my reading of it would say yes.

</comment></comment><comment><USER12> If you want business-friendly maps, Microsoft's MapPoint terms were very startup-friendly last time I checked. You just pay as you go. The downside is you need to talk to a salesguy: <http://www.microsoft.com/maps/>

</comment><comment><USER13> This very issue stopped me dead in my tracks when quoting for a recent project. I was put in touch with an Australian sales manager, from a 3rd party acting on behalf of google (this made me wary)

All communications mentioned 10k starting cost, with a final cost given when I provided detailed project objectives. This was not an option, as the project was confidential.

Suffice to say, I walked away from the project as I could not provide a cost to the client without allowing detailed project info to be passed on to a 3rd party. Also it seemed they were going to assess the project to see how much they could get away with charging me.

They put me in touch with : <http://dmsbt.com/>

</comment><comment><USER14> I was actually surprised by this and I'm sure there are plenty of intranet or small scale subscription services using Google Maps without paying.

Does Microsoft or Yahoo offer more startup friendly terms?

</comment><comment><USER15> The problem is that it's such a huge jump from $0 to $10k. That's the part that feels like a "gotcha". It's also just a huge number, even for a company with funding.

<comment><USER16> Google has the same issue with the Google Apps for your domain. The basic version is free (except for domain name registration fees) for 50 users with similar storage/features as gmail. But the business version is $50/user/year. The costs can very easily escalate into thousands.

<comment><USER17> I would argue that if your company has 50 people, you should already be budgeting for infrastructure, and the sudden jump in costs shouldn't take you by surprise. And $50/user/yr is still a hard-to-beat price if you're making good use of the package.</comment></comment></comment>


Ask HN: How to approach testing in Golang? - <AUTHOR> I am a bit lost when it comes to unit testing in Golang. When developing in Java and Javascript it is easy to mock out data and functions, make assertions and expectations such as making sure some function was called inside a function your testing. What are some things to make sure to test when unit testing Golang functions and what is a good strategy for mocking things out? <comment><USER2> Ha! There are maaaany opinions in this thread, but I will add mine.

I'm not a fan of writing stupid mocks. That doesn't give me much confidence, and spending time on tests is mostly about building ones confidence. I very much prefer running the program end-to-end with defined inputs, and checking its outputs.

Whether that's database, file on disk, or REST api. I write python nosetests that prepare the setup, run golang program, run the checks, and tear down the setup. Sadly, normally it's hard to get golang code coverage for end-to-end tests (aka integration tests). This is what I usually do:

[https://blog.cloudflare.com/go-coverage-with-external- tests/](https://blog.cloudflare.com/go-coverage-with-external-tests/)

</comment><comment><USER3> Take a look at the Golang source for some examples of table-driven tests. When you need mocking you can usually use interfaces and then define a struct in your test that satisfies the methods of the interface.

I'm currently co-authoring a book called Production Go which includes a chapter on testing. That chapter is available in the free sample: [https://leanpub.com/productiongo/](https://leanpub.com/productiongo/) and might cover some of your questions.

</comment><comment><USER4> We make use of [https://github.com/onsi/ginkgo](https://github.com/onsi/ginkgo) and Gomega [https://github.com/onsi/gomega](https://github.com/onsi/gomega) for BDD testing with more expressive matchers. You might be familiar with some of these concepts from Cucumber/Jasmine/Mocha.

They get significant use in Cloud Foundry and in the Kubernetes e2e tests.

I appreciate a lot of people like the simplicity of the standard test library and if you're starting with Go development, it's not a bad idea to get used to that before exploring external dependencies.

Caveat: While not necessarily a "maintainer", I have been doing a bunch of maintenance on these projects the past few weeks.

Edit: Also we use counterfeiter [https://github.com/maxbrunsfeld/counterfeiter](https://github.com/maxbrunsfeld/counterfeiter) for fakes.

</comment><comment><USER5> I like to use a pattern like this: given a resource defined as an interface:

        type Database interface {
            Users() ([]*User, error)
        }

I like to create an explicitly-mocked version of it like this:

        type MockDatabase struct {
            UsersStub func() ([]*User, error)
        }

        func (m MockDatabase) Users() ([]*User, error) {
            if m.UsersStub == nil {
                panic("MockDatabase.UsersStub is nil")
            }
            return (m.UsersStub)()
        }

Then you can create the resources you need, and define how they work, at the top of your test:

        func TestSomething(t *testing.T) {
            db := MockDatabase{
                UsersStub: func() ([]*User, error) {
                    ...
                }
            }

            // use db here
        }

This works well as long as the things you're testing take all the resources they need as parameters.

<comment><USER6> I also use that same pattern. I'm very tempted to write a tool to be called by `go generate` to automatically create that kind of mock.

<comment><USER7> Please don't. I mean, do, whatever makes you happy. There's at least 3 of these flying around in various stages of abandonment.

Here's one:

[https://github.com/vektra/mockery](https://github.com/vektra/mockery)

<comment><USER6> Do any of those build this style of mock? I looked around a while back and didn't come across one.

 _edit_ : I looked again and found [https://github.com/matryer/moq](https://github.com/matryer/moq) which seems similar

</comment></comment></comment></comment><comment><USER8> Design components based on interfaces. Use dependency injection, Testify [0] for asserts and mocks, and Mockery (based on Testify) [1] to generate mocks.

[0] [https://github.com/stretchr/testify](https://github.com/stretchr/testify) [1] [https://github.com/vektra/mockery](https://github.com/vektra/mockery)

</comment><comment><USER9> I found the talk "Advanced Testing With Go" to be illuminating.

Video: [https://www.youtube.com/watch?v=yszygk1cpEc](https://www.youtube.com/watch?v=yszygk1cpEc)

Slides: [https://speakerdeck.com/mitchellh/advanced-testing-with- go](https://speakerdeck.com/mitchellh/advanced-testing-with-go)

</comment><comment><USER10> Java and co can generate logic at runtime, go cannot so you have to manually write mocks, or "generate" them with a third party executable.

You cannot mock anything if you don't use go interfaces at first place, there is no inheritance in go so you can't swap types unless they are interfaces or convertible.

given an interface Fooer :

        type Fooer interface {
           Do(int)int
        }

and a method to test

        func Accept(f Fooer){
           f.Do(10)
        }

it's easy to write a mock

        type FooImpl struct {
          T testing.T
        }

        func(f FooImpl)Do(i int)int{
          f.T.Helper()
          if i!=10{
             f.T.Fail("i=10 expected")
          }
          return 0
        }

        // the test
        func TestAccept(t testing.T){
           Accept(FooImpl{T:t})
        }

Again, your methods have to accept interfaces at first place or you can't mock anything. With Go you'll often have to step back and do manually the things you took for granted in Java.

Only go maintainers can fix the "issue", by allowing the method definition on ad hoc structs, which is impossible right now.

</comment><comment><USER11> Also, consider what you are really testing with mocks. Your implementation, or your mock? You can test without mocks..,

<comment><USER12> A mock is a fake. Just so we are using the same terminology I'll define my terms:

Fake: a test implementation of something that stands in for production code to solve certain testing problems.

A fake can implement an API and provide simplified behaviour. A good example might be a piece of code that stands in for a hardware device driver that you use when you don't have access to the hardware. Another very common use of a fake is to fake out a UI layer. There are lots of other common places for pure fakes.

There are 2 special kinds of fakes:

\- Stub: An implementation that returns canned data.

\- Mock: An implementation that encapsulates an expectation.

The most common type of mock is a mock that encapsulates the expectation that a certain function is called.

Sometimes you don't want to write a pure fake because it's more than you need. Most of the time you can adequately test what you want without talking directly to some inconvenient real API with a stub.

There are many good reasons to use pure fakes or stubs. Like I mentioned, it might be because hardware is involved. Sometimes it's because you are using some SaaS that you don't want to really access in your tests. Very often it's because the real service is just too slow (I'll need another post to describe why this is super important, but even if you don't agree I hope you will see that it's not the only reason to use a fake or a stub).

As soon as you use a fake or a stub, though, you leave a hole in your tests -- you have no way of knowing that the production code uses the _real_ service in the production code. This is when you need a mock. You can go without testing it, but mocking it in this circumstance is almost always a better solution.

Of course, there is a school of thought that reaches for mocks _first_. This school of thought is often referred to as the "London School" because it is very popular (and probably originated from) several very famous people who work in London. The GOOSE book, which describes outside-in programming is probably the best description of this school of thought. I've met quite a few of the people who are standard bearers for this school of thought and they are very talented developers. Again, I would need another post to describe why I don't think it's the best technique to reach for first, but it's an area where reasonable people can disagree.

Outside-in and promiscuous mocking has several advantages -- especially when you are less experienced. It provides a framework for reasoning about how to do design. Personally, I am not a big fan in general, but there are times I use it -- I just replace the mocks with non-faked tests after the fact.

So, while I agree that one should have the mindset to use mocks as a technique of last resort, it's still a valuable tool in your arsenal.

<comment><USER13> >Outside-in and promiscuous mocking has several advantages -- especially when you are less experienced. It provides a framework for reasoning about how to do design.

I find that _any_ additional couplings to your code makes the couplings you do have more painful to work with.

Some people think that by adding this kind of 'tight coupling' \- low level mocks / unit tests - you feel that pain earlier on in the design process and that pain drives you to make a better, more loosely coupled code design.

Personally, I think that this approach is rather like forcing kids to smoke cigarettes every day to show them how much of a filthy dirty habit it is.

<comment><USER12> Very similar to cigarettes, the really nasty effects don't usually materialise until quite far down the road -- when you don't have much opportunity to deal with them.

However, I don't think it _has_ to be like this. I sometimes call mocking "wish based design". Sometimes you don't really know what shape is going to be good. You know ahead of time that TDDing a solution is likely to churn a lot of time writing stuff you are going to ultimately throw away. You can spike a solution to get some more information, but there is often a lot of pressure to ship whatever you spiked, no matter how successful you were (or how bad your tests ended up being).

With a mocking approach you think to yourself, "If this were already written and I was using it, ideally how would it work?" You mock your wish and you write the use side of the code in your tests. Quite frequently this reveals most of your naivety and allows you to design what it is you need. At that point, you replace the mocks with real implementation. It's at this point where I differ from the main proponents of the style (or at least last time I talked to them, which is admittedly several years ago). I will TDD my implementation, removing the mocks as I go.

So, on reflection, I guess I use it in places where I would otherwise spike a solution. Again, for fairly junior people, it can be a great technique to help them understand where to start. Often I find that junior people can't do TDD because they simply can't envision what they are building. They will spike a solution, jam a couple of tests in to show that it's basically working and call it a day. A mocked solution often ends up exposing state in the places where it needs to be. This is frequently better than the spiked solution which is usually an encapsulated ball of mud.

As I said before, though, I use it as a technique of last resort, not a technique of first resort.

<comment><USER13> >However, I don't think it has to be like this. I sometimes call mocking "wish based design". Sometimes you don't really know what shape is going to be good.

That's why I write executable specs at a high level with my "wish based design", make that "test" pass and then refactor. Wish first -> code next -> only _well designed_ code after that.

I don't feel the need to dive down another level and mock out real modules that are already covered by this high level test. That's just creating unnecessary work for myself and introducing a potential source of test bugs (where the mock doesn't match the reality).

</comment></comment></comment></comment></comment><comment><USER14> Check out [https://github.com/stretchr/testify](https://github.com/stretchr/testify) for more convenient assertions, as well as mocking support for implementing interfaces within your tests.

You cannot do ad-hoc mocking of arbitrary functions in Go. If you want to write component-level tests that mock out other components, then you need an interface boundry between those components. Or better yet, you just write tests at the network level, spinning up mock clients/servers on localhost.

<comment><USER15> I find vanilla Go testing (if res != exp { t.Fatal(...) }) much more versatile than testify. Testify did not cover some of my mocking use cases, and response on the repo was tepid, so I ditched it and was happier (this is a user POV, please don't come bashing me on not contributing to open source, etc.)

For mocks, I use GoMocks ([https://github.com/golang/mock](https://github.com/golang/mock)), again a "standard" lib of go. It has Expect syntax where you can assert that mocks were called in a certain way.

I use `go generate` to generate my mocks via interfaces. You should be comfortable with abstracting your specific implementation with interfaces. It is like the first must-do step to Go testing, since we cannot monkey patch or duck type stuff on the fly unlike in JS or Ruby... Second step is to generate mocks from the interface, and inject these dependencies into say your API HTTP Handler... Third step is to assert that your mocks were called in a certain way, and you can also craft different return values from your mocks, to test different scenarios. I also extended some of the gomock Matchers to be able to return different stuff on the fly for e.g in the case of row.Scan(&ptrToInt, &ptrToString), etc.

Ultimately I think this is the Golang ethos: Stick to the standard way of doing things, even if you are allergic to it at first. You will be happier.

<comment><USER16> Not a good example (if res != exp { t.Fatal}) is almost entirely require.Equal in testify (the message is logged that you specify, and the test is immediately stopped).

[https://github.com/stretchr/testify/blob/master/require/requ...](https://github.com/stretchr/testify/blob/master/require/require.go#L90)

I get a lot of the rationale behind the standard way ethos. I think this is just a bad example of it. Tests and lack of good OOB patterns for passing up error information being two of my primary issues with the "Go" way (and it only intensifies over time), but that's just my feelings and I get I'm in the minority (though Cheney seems to have a lot of the same issues I have with the standard error package [https://dave.cheney.net/2016/04/27/dont-just-check- errors-ha...](https://dave.cheney.net/2016/04/27/dont-just-check-errors- handle-them-gracefully))

</comment></comment></comment><comment><USER17> If you're comfortable with this style of testing, you can do it in Go using interfaces and code generation.

Counterfeiter hooks into `go generate`: [https://github.com/maxbrunsfeld/counterfeiter](https://github.com/maxbrunsfeld/counterfeiter)

Here's an example project: [https://github.com/andreasf/cf-mysql- plugin](https://github.com/andreasf/cf-mysql-plugin)

</comment><comment><USER18> If you want to make testing a bit less verbose, you can look into testify ([https://github.com/stretchr/testify](https://github.com/stretchr/testify)), it adds some helpful assertions (no more 'if err != nil ...').

If you are using interfaces with dependency injection, it is easy to mock the dependencies of whatever you are unit testing implementing the interfaces you have defined in your testing file.

</comment><comment><USER19> A colleague of mine made a tutorial for GoMock, maybe its helpful: [https://blog.codecentric.de/en/2017/08/gomock- tutorial/](https://blog.codecentric.de/en/2017/08/gomock-tutorial/)

</comment><comment><USER20> It's pretty much the same as in other languages, so don't over think it. The main difference for you will be Go is compiled, so you have to think slightly differently when writing tests.

Here are a few don'ts:

\- don't take interfaces as arguments unless it really makes sense, even if it makes testing easier

\- don't use one of the tiny test helper libraries that basically just writes if statements for you

\- don't start writing tests until you've already figured out how to organize your logging and errors

One pattern I found helpful was to write an adapter that uses the testing.T.Print* functions as the destination for log.Print* statements in my regular code so that log messages aren't jumbled when running tests in parallel.

</comment><comment><USER21> 1\. Interfaces allow swapping out objects.

2\. Mocking is a bad idea even in languages that support it well.

A better approach, when you do need to deal with fakes, is verified fakes: write two versions of an interface, and a test suite for that interface to ensure the fake acts the same as the real thing.

More here: [https://codewithoutrules.com/2016/07/31/verified- fakes/](https://codewithoutrules.com/2016/07/31/verified-fakes/)

</comment><comment><USER22> I have recently created a small project called rocket ([https://github.com/joernlenoch/rocket](https://github.com/joernlenoch/rocket)) to provide myself with a simple dependency injection module for exactly this reason.

This is rather a starting point than a solution for your problem. But combined with some "fakers" and supplied testing frameworks this might remove some of the boilerplate.

</comment><comment><USER23> The age of Docker means IMO you should avoid mocks. There's no reason not to have a full integration test against your real API especially because you can compile and run test binaries independently.

<comment><USER24> Not an apologist for unit tests, but I'll bite here: unit tests are still valuable when you have the capacity to run ful integration tests.

A unit test can run really fast, for one, it can cover future use cases that don't get covered in integration, it edge cases you don't expect to hit, but are still important to cover in case something happens.

I'm sure there are more, good reasons to write unit tests _and_ integration tests, but those two easy pickings are probably the source of your down votes

<comment><USER25> Unit tests run much faster and are more focused - relying too much on integration tests to catch bugs is a massive maintenance smell, as you will waste an enormous amount of time trying to figure out the root cause of a failing test.

</comment></comment></comment><comment><USER26> There's a good JustForFunc episode on testing. [https://www.youtube.com/watch?v=hVFEV- ieeew](https://www.youtube.com/watch?v=hVFEV-ieeew)

</comment><comment><AUTHOR> Thank you everyone for your feed back I am going to take this and digest it

</comment><comment><USER27> I'm using table tests so that everything stays clean, and Vectra mockery for generating mocks and doing assertions.

Works great so far.</comment>


Google Maps API reduced their cost by 94% - <AUTHOR> https://blog.cityflo.com/how-we-reduced-our-google-maps-api-cost/ <comment><USER2> click bait title. "How we reduced our Google Maps API cost by 94%" is the actual title</comment>


How to Store Data Forever - <AUTHOR> https://drewdevault.com/2020/04/22/How-to-store-data-forever.html <comment><USER2> If you think others might be interested in helping out, creating+seeding a torrent and pinging r/DataHoarder [0] is one of the most effective ways of getting redundancy for large amounts data. It's a subreddit for people who have massive storage setups in their homes (think tens/hundreds of terabytes). Besides "Linux ISOs", they've helped store threatened YouTube channels and government data in case of deletion.

[0]: [https://old.reddit.com/r/datahoarder](https://old.reddit.com/r/datahoarder)</comment>


Glob Matching Can Be Simple and Fast Too - <AUTHOR> https://research.swtch.com/glob <comment><USER2> _I have not looked at the other linear-time implementations to see what they do, but I expect they all use one of these two approaches._

Python's glob() (fnmatch really) translates the glob to a regular expression then uses its re library:

[https://github.com/python- git/python/blob/715a6e5035bb21ac49...](https://github.com/python- git/python/blob/715a6e5035bb21ac49382772076ec4c630d6e960/Lib/fnmatch.py#L72)

<comment><USER3> Thanks for pointing this out. I wrote a Python test program but forgot to run that test in the data collection for the graphs. I will update them once the tests finish running. Ironically, Python must be passing the glob to an exponential-time regexp library, since Python is on the slow side of the fence.

<comment><USER4> Russ, while you're around, thanks for re2! RE2::Set especially! But why is RE2::Set so underdocumented and underpromoted? All regex libraries need this functionality.

<comment><USER3> You're welcome. I think basically all of RE2 is underdocumented and underpromoted. Why should RE2::Set be any different?

Seriously, though, it was a bit of an afterthought. A team at Google was |ing together a ton of regexps and came to me for something better, so I wrote RE2::Set. I'm glad it helps others too.

</comment></comment></comment></comment><comment><USER5> Previously: "Regular Expression Matching Can Be Simple And Fast" (2007) [https://swtch.com/~rsc/regexp/regexp1.html](https://swtch.com/~rsc/regexp/regexp1.html) The paper deals with "Thompson NFA" approach to regex, with low computational complexity.

Other Russ' papers on regular expression matching: [https://swtch.com/~rsc/regexp/](https://swtch.com/~rsc/regexp/)

<comment><USER6> It's funny, I had an immediate "I've seen these graphs before" reaction" with this article, and it turns out your first link is something I was working on last month for a leetcode question.

It was probably one of the more useful/understandable pages I found in that process.

</comment></comment><comment><USER7> Interesting how the Rust implementation of glob currently seems to be the slowest out of the linear time implementations. I guess maybe not too much optimisation effort was put into it?

<comment><USER8> There's no implementation of globbing in Rust's standard library, so we'd need more information regarding which implementation was used for the blog post. I'm guessing one of two possibilities:

1\. The glob crate, which is stdlib ejecta from pre-1.0, and appears to be in maintenance mode (it hasn't seen a "real" commit since Jan 2016).

2\. The regex crate, which is very actively developed (and in fact inspired by rsc's writings and Go's regex implementation).

If it's the former, then indeed it's probably as disinteresting as "nobody has bothered to ever benchmark this crate". But if it's the latter then I bet burntsushi would be very interested!

<comment><USER9> On a glance, I don't see any support for globbing in regex crate.

<comment><USER10> I guess it's the one that's part of ripgrep,

[https://github.com/BurntSushi/ripgrep/tree/master/globset](https://github.com/BurntSushi/ripgrep/tree/master/globset)

> This crate implements globs by converting them to regular expressions, and > executing them with the regex crate.

</comment></comment></comment></comment><comment><USER11> There's another way for glob() implementations to mitigate these sort of patterns that Russ doesn't discuss, but can be inferred from a careful reading of the different examples in this new glob() article & the 2007 regex article.

In the regex article he notes that e.g. perl is subject to pathological behavior when you match a?^na^n against an a^n:

        $ time perl -wE 'my $l = shift; my $str = "a" x $l; my $rx = "a?" x $l . $str; $str =~ /${rx}/' 28
        real    0m13.278s

However changing the pattern to /${rx}b/ makes it execute almost instantly. This is because the matcher will look ahead for fixed non-pattern strings found in the pattern, and deduce that whatever globbing we're trying to match now it can't possibly matter if the string doesn't have a "b" in it.

I wonder if any globbing implementations take advantage of that class of optimization, and if there's any cases where Russ's suggested solution of not backtracking produces different results than you'd get by backtracking, in particular with some of the extended non-POSIX glob syntax out there.

<comment><USER12> That's not globbing but using the regex matcher. perl's glob does the same as PHP, it simply calls the system glob. So I'm curious why the graph displays it as exponential on linux, where it should be linear.

In your example pcre2 performs much better than perl btw: it errors with match limit exceeded (-47), while perl happily burns exponential CPU. It's now even worse than before Russ' original perl article. Now it descends into heap memory, before only into the stack. So now it will keep crunching forever on the whole heap, while before the perl 5.10 rewrite triggered by Russ it died fast on stack overflow.

<comment><USER3> Empirically, Perl's glob does not call the system glob. On my Linux system Perl (5.18.2) is slow but the system glob is fast. Create a file named /tmp/glob/$(perl -e 'print "a"x100') in an otherwise empty /tmp/glob and then try:

      $ cat tglob.pl
      #!/usr/bin/perl

      use Time::HiRes qw(clock_gettime);

      $| = 1;
      chdir "/tmp/glob" || die "$!";
      for($i=0; $i<9; $i++) {
          $pattern = ("a*"x$i) . "b";
          $t = clock_gettime(CLOCK_REALTIME);
          $mul = 10;
          if($i >= 5){
              $mul = 1;
          }
          for($j=0; $j<$mul; $j++) {
              glob $pattern;
          }
          $t1 = clock_gettime(CLOCK_REALTIME);
          printf("%d %.9f\n", $i, ($t1-$t)/$mul);
      }
      $ perl -v

      This is perl 5, version 18, subversion 2 (v5.18.2) built for x86_64-linux-gnu-thread-multi
      ...
      $ perl tglob.pl
      0 0.000004911
      1 0.000016212
      2 0.000088072
      3 0.002416682
      4 0.030226517
      5 0.452545881
      6 6.872966528
      ^C

You're the second person to claim that Perl calls the system glob though (someone in my blog comments did too). Maybe different versions of Perl do different things? This is Ubuntu 14.04 if that matters.

<comment><USER13> No you are right. Modern perls use the BSD code in File::Glob.

<comment><USER3> More discussion at [https://research.swtch.com/glob#comment-3272455315](https://research.swtch.com/glob#comment-3272455315)

</comment></comment></comment></comment></comment><comment><USER14> OP, what version(s) of the BSD libc did you test? What OS, which version of the OS.

macOS only? NetBSD? FreeBSD? OpenBSD?

If you tested on FreeBSD, please file a bug at [https://bugs.freebsd.org/bugzilla/enter_bug.cgi?product=Base...](https://bugs.freebsd.org/bugzilla/enter_bug.cgi?product=Base%20System)

I'm not a project member but I'm a user of the system so it's in my interest that issues like this are resolved.

Please let me know whether or not you file a bug so that if you do I don't duplicate bug reports and if you don't I can do some benchmarking myself.

<comment><USER3> I copied the glob implementation from one of the BSDs - I believe FreeBSD - into a standalone C program and ran that program on the same Linux system as the rest of the tests. Here's the version that tests the system glob. If you run it on your FooBSD systems you can see whether it runs quickly or not. The program assumes that you've already done:

      rm -rf /tmp/glob
      mkdir /tmp/glob
      cd /tmp/glob
      touch $(perl -e 'print "a"x100')

And here's the program:

      #include <stdio.h>
      #include <glob.h>
      #include <unistd.h>
      #include <string.h>
      #include <stdlib.h>
      #include <dirent.h>
      #include <time.h>

      int
      main(void)
      {
          glob_t g;
          char pattern[1000], *p;
          struct timespec t, t2;
          double dt;
          int i, j, k;

          chdir("/tmp/glob");
          setlinebuf(stdout);

          int mul = 1000;
          for(i = 0; i < 100; i++) {
              p = pattern;
              for (k = 0; k < i; k++) {
                  *p++ = 'a';
                  *p++ = '*';
              }
              *p++ = 'b';
              *p = '\0';
              printf("# %d %s\n", i, pattern);
              clock_gettime(CLOCK_REALTIME, &t);
              for (j = 0; j < mul; j++) {
                  memset(&g, 0, sizeof g);
                  if(glob(pattern, 0, 0, &g) != GLOB_NOMATCH) {
                      fprintf(stderr, "error: found matches\n");
                      exit(2);
                  }
                  globfree(&g);
              }
              clock_gettime(CLOCK_REALTIME, &t2);
              t2.tv_sec -= t.tv_sec;
              t2.tv_nsec -= t.tv_nsec;
              dt = t2.tv_sec + (double)t2.tv_nsec/1e9;
              printf("%d %.9f\n", i, dt/mul);
              fflush(stdout);
              if(dt/mul >= 0.0001)
                  mul = 1;
              if(i >= 8 && dt/mul >= 10)
                  break;
          }
      }

I won't be filing any specific bugs myself. I mailed oss-security@ this morning, which should reach relevant BSD maintainers, but more bug filing can't hurt.

</comment></comment><comment><USER11> Slightly off-topic, but anyone know what he's using to generate those inline SVG graphs? I've been looking for some easy to use graphing library like that to present similar performance numbers on a webpage.

<comment><USER3> I looked around but didn't find anything I liked. The regexp article [1] uses jgraph [2] to plot the data as eps, then ghostscript to turn eps to png. That no longer works in a world of variable-dpi screens, so for this article I generated SVGs and inlined them into the HTML. The SVGs are generated by a custom program I wrote for the job [3], to mimic the old jgraph graphs. It's not pretty code, but it gave me a lot of control over the presentation and produced decent results. You're welcome to adapt it if you like.

[1] [https://swtch.com/~rsc/regexp/regexp1.html](https://swtch.com/~rsc/regexp/regexp1.html)

[2] [https://web.eecs.utk.edu/~plank/plank/jgraph/jgraph.html](https://web.eecs.utk.edu/~plank/plank/jgraph/jgraph.html)

[3] [https://research.swtch.com/globgraph.go](https://research.swtch.com/globgraph.go)

</comment></comment><comment><USER15> Not sure if OP is author, but if you are, just to inform you, there is a small typo in this paragraph:

"Unfortunately, none of _tehse_ protections address the cost of matching a single path element of a single file name. In 2005, CVE-2005-0256 was issued for a DoS vulnerability in WU-FTPD 2.6.2, because it ran for a very long time finding even a single match during:"

Very informative article. Thanks for it!

<comment><USER3> Thanks, fixed the typo (author, not OP).

</comment></comment><comment><USER16> The bsd derived glob has other functionality that I assume isn't simple or fast:

      perl -MFile::Glob=bsd_glob -e 'print bsd_glob("{{a,b,c}{1,2,3}{{yuck,Yuck},{urgh,URGH}}}\n")'

Produces 36 lines representing all the iterations. Nest a bit deeper and it gets unwieldy.

<comment><USER17> Those are actually expanded (in the recursive fashion you would expect) before any "star" matching is done.

</comment></comment><comment><USER18> I wonder whether it would help to match from both sides (start and end) simultaneously, since you know you're not looking in the middle of the string. You also don't care about capture groups.

<comment><USER3> It would help this example. It wouldn't help the general case.

<comment><USER18> But in general the reversal of a glob pattern is trivial while the reversal of the equivalent regular expression is not, no?

<comment><USER3> Reversing a regular expression and reversing a glob are about the same: you just flip everything around and you're done. In fact RE2 uses this fact to speed up regexp searches. See [https://swtch.com/~rsc/regexp/regexp3.html#submatch](https://swtch.com/~rsc/regexp/regexp3.html#submatch) and scan ahead a bit for "Run the DFA backward".

</comment></comment></comment></comment><comment><USER19> For fun, I ran this against node-glob ( [https://github.com/isaacs/node- glob](https://github.com/isaacs/node-glob) ).

Looks like it exhibits the slower behavior:

      n,elapsed
      1,0.07
      2,0.07
      3,0.07
      4,0.07
      5,0.16
      6,1.43
      7,19.90
      8,240.76

See this gist for the script [https://gist.github.com/mixu/e4803da16e42439480eba2b29fa4448...](https://gist.github.com/mixu/e4803da16e42439480eba2b29fa44484)

</comment><comment><USER20> > _Graphical FTP clients typically use the MLST and MLSD commands_

Do not count WWW browsers amongst the number of those graphical FTP clients. The common WWW browsers that speak FTP use LIST or LIST -l . With the exception of Google Chrome when it thinks that it is talking to a VMS program, they do not pass pattern arguments, though.

</comment><comment> libre-man I tested Common Lisp. SBCL seems to be exponential while Clozure CL is not.

However it should be noted that it is non portable to do globbing in Common Lisp, so I expect most users implement it using something CL-FAD or OSICAT and CL-PPCRE, and CL-PPCRE is efficient.

</comment><comment><USER21> I've been playing around with my own glob implementation. From what I've seen, the simplified algorithm mentioned in the article wouldn't be able to handle question marks. In particular, I don't think a non-backtracking algorithm can handle a pattern like "?a _?a_?a _?a_?b". I've been working to minimize the worst-case behavior, but it's tricky.

<comment><USER20> Your semantics for question marks are wrong. Question marks matching zero or one characters are the semantics for IBM/Microsoft command interpreters, for reasons that go back all of the way to CP/M. (Strictly speaking the original semantics amounted to, because of the 8.3 field padding, question marks matching a character, the end of the string, or zero characters if immediately before the dot.)

In POSIX, question marks match _exactly one_ character, always. There's no need for backtracking.

* [http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3...](http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_13_01)

<comment><USER21> HN screwed up my comment. It's not supposed to be "question mark, a, question mark, a, ...". I meant to write "question mark, a, star, question mark, a, star, ...".

<comment><USER20> That would be backtracking caused by those stars, _not_ by the question marks. Hacker News didn't screw up the part where you wrote "wouldn't be able to handle question marks".

<comment><USER21> A pattern like "star a star a star a star" can be handled by the algorithm described in the article:

> Consider the pattern "a star bx star cy star d". If we end the first star at > the first bx, we have the rest of the name to find the cy and then the d. > Using any later bx can only remove choices for cy and d; it cannot lead to a > successful match that using the first bx missed. So we should implement a > star bx star cy star d without any second-guessing, as "find a leading a, > then find the earliest bx after that, then find the earliest cy after that, > then find a trailing d, or else give up."

This algorithm doesn't work if the pattern has question marks.

<comment><USER3> Yes it does.

</comment></comment></comment></comment></comment></comment><comment><USER22> Sorry, but the implementation posted is O(|pattern| * |name|), not linear. [http://ideone.com/2xCXyY](http://ideone.com/2xCXyY)

<comment><USER23> The size of the pattern is held as constant. Or more accurately, "it's linear" is an abbreviated form of "it's linear in the size of the text searched."

</comment></comment><comment><USER24> How about the default sort? Ouch or no ouch?

</comment><comment><USER25> We independently reinvented an adaptation of this algorithm for Monte's "simple" quasiliteral, which does simple string interpolation and matching. The code at [https://github.com/monte- language/typhon/blob/master/mast/pr...](https://github.com/monte- language/typhon/blob/master/mast/prelude/simple.mt#L68-L121) is somewhat similar in appearance and structure to the examples in the post.

      def name := "Hackernews"
      # greeting == "Hi Hackernews!"
      def greeting := `Hi $name!`
      # language == "Lojban"
      def `@language is awesome` := "Lojban is awesome"

A quirk of our presentation is that adjacent zero-or-more patterns degenerate, with each subsequent pattern matching the empty string. This mirrors the observation in the post that some systems can coalesce adjacent stars without changing the semantics:

      # one == "", two == "cool"
      def `adjacent @one@two patterns` := "adjacent cool patterns"

<comment><USER26> hey BuuQu9hu, your comment started out as [dead] for some reason. You may have been inadvertently hellbanned.

<comment><USER27> ...hellbanned?

<comment><USER28> Being (silently) hidden so that your posts look okay to you but don't show up for others (unless they have 'show dead' set to on).

It's an effective filtering mechanism, but some consider it rather cruel and try to notify posters about it.

<comment><USER26> False positives suck. There are people coming on here day after day, posting valuable or at least worthy comments, and nobody can see them. They waste their time unknowingly. It is cruel.

</comment></comment></comment></comment></comment><comment><USER29> Why write a glob engine at all when you already have a fast regex implementation that can match both exact paths and plausible subtrees?

The bulk of the haskell code to do this:

        parseGlob :: Char -> Char -> String -> Parser Glob
        parseGlob escC sepC forbid =
            many1' (gpart <|> sep <|> glob <|> alt) >>= return . GGroup . V.fromList
          where gpart = globPart escC (sepC : (forbid ++ "{*")) >>= return . GPart
                sep = satisfy (== ch2word sepC) >> return GSeparator
                alt = do
                  _ <- AttoC.char '{'
                  choices <- sepBy' (GEmpty `option` parseGlob escC sepC (",}" ++ forbid)) (char ',')
                  _ <- AttoC.char '}'
                  return $ GAlternate $ V.fromList choices
                glob = do
                  res <- takeWhile1 (== ch2word '*')
                  if B.length res == 1 then
                    return GSingle
                  else
                    return GDouble

        wrapParens s = T.concat ["(", s, ")"]

        globRegex :: Char -> Glob -> T.Text
        globRegex sep  GSingle       = T.concat ["([^", T.singleton sep, "]*|\\", T.singleton sep, ")"]
        globRegex _    GDouble       = ".*"
        globRegex _    GEmpty        = ""
        globRegex sep  GSeparator    = T.singleton sep
        globRegex sep (GRepeat a)    = T.concat ["(", T.concat (V.toList $ fmap (globRegex sep) a), ")*"]
        globRegex sep (GGroup a)     = T.concat $ V.toList $ fmap (globRegex sep) a
        globRegex _   (GPart p)      = T.concatMap efun base
          where base = TE.decodeUtf8 p
                escChars = S.fromList ".[]()\\{}^$*+"
                efun c = if S.member c escChars
                         then T.concat ["\\", T.singleton c]
                         else T.singleton c
        globRegex sep (GAlternate a) =
            if V.null alts
            then ""
            else T.concat [altsStr, if hasEmpty then "?" else ""]
          where hasEmpty = isJust $ V.find (== GEmpty) a
                alts = fmap (globRegex sep) $ V.filter (/= GEmpty) a
                altsStr = wrapParens $ T.intercalate "|" $ V.toList alts

<comment><USER30>
        Why write a glob engine at all when you already have a fast regex

you didn't read the article. there are glob implementations that do just that.

<comment><USER29> The bulk of the article is about how the Go implementation avoids this behavior without just converting to regex.

<comment><USER31> No, the Go standard library implementation is mentioned in a single paragraph. The Go code in the article is written simply as an illustration.

</comment></comment></comment></comment><comment><USER32> [https://github.com/skarnet/execline/raw/master/src/execline/...](https://github.com/skarnet/execline/raw/master/src/execline/elglob.c)

[https://github.com/skarnet/execline/raw/master/src/libexecli...](https://github.com/skarnet/execline/raw/master/src/libexecline/exlsn_elglob.c)

Simple.

[http://www.in-ulm.de/~mascheck/various/argmax/](http://www.in- ulm.de/~mascheck/various/argmax/)

       execlineb -c 'elglob a /*/*/*/* ls $a'

(statically-linked execlineb)

If I am not mistken, ARG_MAX will be the limit.

Straightforward.

<comment><USER20> That's a wrapper around the C library's glob() function, which is what the headlined article was looking at. What point are you trying to make?</comment></comment>


Google jukes the stats - <AUTHOR> http://www.marco.org/2013/04/03/google-jukes-the-stats <comment><USER2> As market share statistics, he's right they are now bunk. But they are on developer.android.com, not in a press release or a company performance report.

As a developer these statistics are a lot more valuable, since they reflect the actual market for Android users shopping for my apps.

I don't think Google is really all that worried about what people think.</comment>


HippyVM - <AUTHOR> http://hippyvm.com/ <comment><USER2> This keep coming up again and again.

As a PHP perf guy for serveral years, I have some weird questions to ask.

Is this an HTTP serving language VM or just a PHP as a language VM?

Because I spent years fixing the first part without ever really doing anything about the second part of that question.

If it can't listen on port 80, my interest drops a lot in the new VM - I'm not going to write scipy/numba.py problems in PHP, even if I could.

<comment> X-Istence Even if it is the latter, couldn't you write a pure PHP implementation of a web server, and use that to server your PHP website?

Much like WSGI in the Python world?

<comment><USER3> ReactPHP does exactly that, heh. Running an event loop on HippyVM with RPythons JIT? Yes please!

</comment></comment></comment><comment> vhost- I hope that website isn't on HippyVM because it's taking forever to load.

<comment><USER4> > At present it does not include a reasonable web server integration, so it's > not usable for use in production in the current form.

RTFA

<comment><USER5> It's difficult to RTFA when TFA doesn't even load.

</comment></comment></comment><comment><USER6> Very cool that this uses the PyPy toolchain, I hope this becomes a trend because it means we have one make-dynamic-languages-fast effort that will pay dividends across multiple languages.

<comment><USER3> Well, it doesn't surprise me as RPython is bloody amazing. I actually started doing something like HippyVM about six months ago for work, to explore if it was possible, and sure enough someone has gone way further than I did haha

</comment></comment><comment><USER7> I would like to mention my (just released) PHP preprocessor here, since it has a feature that suggests a number of micro-optimizations. People seem to care so much about performance, so why not spend a few minutes to improve source code in the first place?

I wrote about the PHPippo PHP preprocessor a few hours ago, though it went through HN unnoticed (so far): [https://news.ycombinator.com/item?id=7861616](https://news.ycombinator.com/item?id=7861616)

</comment><comment><USER8> As someone with a bit of experience scaling PHP, it seems to me that the main problem with the language is not its raw performance (which seems on a par with its more cousins Python and Ruby) but rather its __single-request model __. For every request, mod_php creates a new interpreter, which has to do all the work to set up the framework, auto-loading the required classes, reading additional data every time. What 's more, the request has to create a new database connection for every request; the single-request model doesn't allow the use of a database pool. (This doesn't change fundamentally with handlers other than mod_php, such as php-fpm.)

With languages like Python and Ruby, not to mention Java, a single interpreter process or thread can handle many requests and can re-use the database connection. In PHP, you're forced to do all the set-up all over again for every request. This is particularly wasteful with frameworks like Laravel, which do a non-trivial amount of work for setting up the application object, IoC controller.

Am I missing something here? I can't believe companies using PHP on a larger scale, like Facebook, discard and re-create the application environment and all DB connections (!) for every single request.

<comment><USER9> That's why you should host using fastcgi, so the PHP processes are long lived. You do pay the startup cost in every request of reconstructing the state (code and session data), but you get the benefit of request isolation. I don't know how other db's handle it, but oracle supports reusing connections across PHP requests.

Facebook does recreate the state on every request, but they've got that down to a 10 ms bootstrap. In my own web services the bootstrapping before the business logic runs is below 20 ms. Most PHP frameworks load too much code up front though, which is why you'll see much worse request bootstrap performance in many cases.

The big benefit of request isolation is predictability. You know exactly what environment your code runs in because you construct it from scratch for every request.

<comment><USER10> "You do pay the startup cost in every request of reconstructing the state..."

And I thought this was the very "problem" that drove people away from CGIs back in the day.

<comment><USER9> It's not a problem, it's a trade-off. An architecture that doesn't keep state across requests trades off per-request performance for stability and scalability, because there is no long-term state on the web server to manage or migrate. CGI traded off too much performance because of the need to launch a process per request, but fastcgi keeps the process alive but throws away the state, so i find the upside beats the downside in a lot of cases (sadly not all, i wish php was less opinionated here, like java where you can choose to have a stateless architecture but don't have to.)

</comment></comment></comment></comment><comment><USER11> PHP is set to be super fast real quick-soon if this HHVM inspired trend continues. A bit like what has been happening to Javascript really..

<comment><USER12> Great to see those languages be rewarded for the quality of their design.

<comment><USER13> Yeah, it's a sad, cruel irony that PHP and Javascript of all languages are getting this sort of extremely focused treatment.

<comment><USER14> Well, already good performing languages like e.g. C++ (and partially Java) don't need this kind of treatment ;-)

</comment></comment></comment></comment><comment><USER15> I wonder what the claimed and "aimed for" compatibility with Zend is actually like. How much of the standard library is implemented? How about common extensions? Is the casting behaviour the same?

Performance is meaningless if your VM can't run people's code.

<comment><USER16> They seem to be using the same test suite as Zend PHP. Which would mean they should be pretty compatible.

Not sure how many of those they are actually passing though. [https://github.com/hippyvm/hippyvm/tree/master/test_phpt/Zen...](https://github.com/hippyvm/hippyvm/tree/master/test_phpt/Zend)

<comment><USER15> Looks like they've copied some of PHP's tests, yes. But being able to run run- tests.php and being compatible aren't quite the same thing, really. It's how many tests pass that count.

Technical side-note: /Zend there appears to be just the tests for the Zend engine (i.e. from PHP's /Zend/tests), while the containing directory contains test folders copied from other parts of PHP (like the standard library)

<comment><USER17> It's also worth noting that not even PHP itself passes their own test suite. At the time of writing [http://gcov.php.net/viewer.php?version=PHP_5_5](http://gcov.php.net/viewer.php?version=PHP_5_5) states:

Overview of PHP_5_5

Build Status: OK Last Build Time: 2 days

Compile Warnings: 611 Code Coverage: 0.3% Test Failures: 78 Expected Test Failures: 42 Valgrind Reports: 44

The "Expected Test Failures" is fascinating. [http://gcov.php.net/viewer.php?version=PHP_5_5&func=expected...](http://gcov.php.net/viewer.php?version=PHP_5_5&func=expected_tests)

<comment><USER15> That's true, but PHP still passes the vast majority of its tests.

</comment></comment></comment></comment></comment><comment><USER18> Does the word "hip" mean something in the context of PHP?

edit: I mean, is there a reason HHVM and HippyVM both use the word "Hip"?

<comment><USER19> [http://en.wikipedia.org/wiki/HipHop_for_PHP](http://en.wikipedia.org/wiki/HipHop_for_PHP)

<comment><USER18> Oh, I should have expanded on my question. I know of HHVM already, I was curious about the use of "Hip" in HHVM and in HippyVm, it appeared that there must be some common heritage

<comment><USER20> I think "Hippy" is just a play on Facebook's "HipHop" VM. Not sure where the latter originated from, though.

<comment><USER18> Thanks, that makes sense. I already assumed that HHVM just happened because it's a cool name. But then seeing the second use of the word made me think there might be something behind it

</comment></comment></comment></comment></comment><comment><USER21> I'm not really familiar with the PyPy tooling and capabilities, but if this takes of, will there be a possibility of interop between RPython language implementations? It would be mind blowing to be able to write libs in different languages and have them working together. We could always use the best one available regardless of the language.

They mention interop with Python code, but I was just wondering if this could go a bit further...

<comment><USER22> As you mention. We have ongoing branch for a implementation of HippyVM-Python bridge. We are hoping to achieve the goal that you described, though we are not sure if this will the last of first step on this path.

</comment></comment><comment><USER23> I got HHVM running wordpress on a relatively high traffic site. The difference is night and day and setup was simple. To have something faster is crazy :)

</comment><comment><USER24> For those that can't find the repository, it can be found here on github [https://github.com/hippyvm/hippyvm](https://github.com/hippyvm/hippyvm) I can never seem to find the link to github from the hippyvm page. It doesn't seem to be very active as far as emerging technology goes, but in my opinion it has good potential since it can build off existing language development tools that are continuously improving.

<comment><USER22> I have to ensure that project under development, 490 commits, not impressive but we are working, here ;)

Though our current problem is which we are trying to fix is ,we have two repos: public and private. We are hopping to solve this soon, and move development to github. Sorry for the confusion.

</comment></comment><comment><USER25> The code look clear, even to me.

Also: Look like it have a debugger! That is something I was looking how do. Plus, bytecode (look clean) too.

RPython could output to ARM?

<comment><USER26> Yes, RPython has an ARM backend.

</comment></comment><comment><USER27> I wonder when they will do a GA instead of pre-release. That would be the right time to test and do benchmarks.

</comment><comment><USER28> So many PHP improvements, PHPNG, HHVM and Hippy, I'm anxious to see the future of PHP :)

</comment><comment><USER29> >It started off as a research project [...] for Facebook

Huh? Isn't that what HipHop is? Can someone elaborate on the matter? Is this a fork? Why aren't they just migrating their work back to HipHop, I mean all this leads to is fragmentation.

<comment><USER26> Facebook had multiple research projects to develop better implementations of PHP programming language. HippyVM is not a fork of HHVM and indeed two codebases are completely unrelated. Therefore "just migrating their work back to HipHop" is not possible; there is nowhere to go back to, since it didn't come from there.

<comment><USER29> But if Hippy is faster why did they pick HipHop?

<comment><USER30> From memory (I think one of the dev's comments on HHVM or Hippy a while back) Hippy started after HHVM was already underway. Hippy was mostly a proof of concept, to see if PyPy would be a viable strategy for a PHP runtime. HipHop was more so the logical followup to HipHop... I think being forked from the HipHop interpreter they were using for development.

</comment></comment></comment></comment><comment><USER31> Older HN thread: [https://news.ycombinator.com/item?id=7459080](https://news.ycombinator.com/item?id=7459080)

</comment><comment><USER32> Can you or someone make one for Ruby please?

<comment><USER16> They are, see Topaz [http://topaz.readthedocs.org/en/latest/](http://topaz.readthedocs.org/en/latest/)

<comment><USER12> "Most of the standard library is missing."

So it'll take some time I guess. Still nice to see people trying, Ruby is a great language but it needs as much love as possible.

<comment><USER33> Or precisely, its VM needs as much love as possible.

<comment><USER12> Or even more precisely, the VM of its main (and de facto default) implementation needs love. Rubinius and JRuby are really VASTLY superior implementations but well, there's no specification so 50% of development time is spent in asinine catch up with an underspecified "standard" (MRI that is).

Really sad state of affairs and the #1 reason why Ruby isn't faster and more feature-packed.

</comment></comment></comment></comment></comment><comment><USER34> haha jokes... well I see why fb wanted to create their own framework since the whole site is in php and it would take em a lot of time to translate it something else (i think most nowadays is JS anyways) but srsly, why people are trying to make faster frameworks for php ... no idea. It doesn't get any faster due its design. At least join one of the open source projects and help that.

Combine all those working on those open source frameworks for years and all together create something fast...

</comment><comment><USER35> I'm a developer and PHP is in my toolbox when suitable. But the thing I can't understand is... why waste all this effort to make a slow, broken language fast, instead of using a faster language?

I mean, sure, it's impressive that this VM is 7 times faster than stock PHP, but Java is over 100 times faster than stock PHP.

What is the gain here? Not having to go out of you little shell and learn a new language?

<comment><USER36> Preexisting codebases, compatibility with great libraries and mature full- featured frameworks, progressive enhancement of infrastructures already in place, recycling of developers. You name it.

This is not about pleasing the bedroom programmer that can just "go out of his little shell and learn a new language". This is great news for any decent sized project or enterprise with infrastructure already running in PHP.

<comment><USER35> > compatibility with great libraries and mature full-featured frameworks

So as I said I've been doing PHP. And I have yet to see one of those unicorns in PHP-land. In fact, my first gut reaction was that you're being sarcastic up there.

<comment><USER36> I was not being sarcastic.

If you already know PHP you should really check out frameworks like Symfony and Laravel (and Symfony Components, amazing reusable pieces to grow your own stuff). Also great tools like Composer for dependency management, PHPUnit, Behat, PHPSpec or Codeception for TDD/BDD. Same with ORMs like Doctrine or the Twig templating engine.

PHP has evolved greatly in the last years. There are tools for most of your needs and you can build stable and robust applications with them, and be excited about the future now that we are seeing efforts like HHVM or HippyVM, or even the internal changes being discussed for PHP 6.

You don't need a new "better" language to make new better products. PHP allows you to write perfectly nice, object oriented, clean code, no problem at all, that integrates easily with proven technologies. It also allows you to write a mess impossible to maintain. If somebody does the later when they can do the former with no extra effort, well, then maybe it's fair to question if that person is a good programmer at all.

<comment><USER35> I'm familiar with Symfony, etc. Please don't sell it to me as "perfectly nice" code.

The reason most of the PHP code is crap is not because PHP makes it impossible, but because a popular crap language with a low initial learning curve tends to attract a lot of crap programmers.

I'd actually say PHP has only gotten better over time, and every new release is better than the previous one. But you have a crap base (which PHP maintains religiously as not to break BC), and a crap community, and the software they write is crap. It's just what it is. The odd exception here and there.

You can overcome a horrible language, but that's not the problem (plus I'd argue PHP isn't the worst out there).

No, it's the crappy community around it. It's hard to overcome the horrible low quality PHP code libraries people write in PHP, because writing everything from scratch isn't practical in any language these days.

Composer is great and all (except for the lack of package signing, hey, who needs security, right?), but all Composer means you can install other people's PHP crap code more conveniently.

And while I wouldn't call the people behind Symfony "crap programmers", that framework is horribly overdesigned, slow, and comprised of an endless Inception of leaky abstractions.

It shows that a programmer can take all their experience and knowledge of design patterns, best practices and what not, and write a complete piece of turd.

Anyway I better run, because the modding of my posts here is showing I've angered the PHP fans.

<comment><USER36> Okay, I get where you are coming from. And I have to say that I agree with you on several of your points. But what I wanted to stress in my sort of "defence" of PHP is that, the fact is, lots of valid programmer use it, either for familiarity reasons, due to existing infrastructures or because they actually enjoy it (I do sometimes!). Complete rewrites are a mythical creature in the real world, so why not embrace, improve and push forward anything that tries to make PHP better in the future, instead of dismissing the whole technology as useless.

</comment></comment></comment></comment></comment></comment><comment><USER37> Waiting for that Perl 6 implementation in RPython any day now.

<comment><USER38> Actually it is more likely to be the a JVM compatible one.

[http://www.infoq.com/presentations/invokedynamic](http://www.infoq.com/presentations/invokedynamic)

</comment></comment><comment><USER39> Seems like an odd way to show off a technology boasting speed when it's cracking under the load. Can anyone confirm if it's running on hippyvm?

<comment><USER11> I can confirm it is not running hippyvm since the page is saying there's no way to use the vm as server language yet (..)</comment></comment>


Common Async/Await Design Patterns in Node.js - <AUTHOR> http://thecodebarbarian.com/common-async-await-design-patterns-in-node.js.html <comment><USER2> I wonder why Promise.all is not a language construct but a regular method call bringing Promise object out of nowhere.

It would've been easier if await just accepted arrays just like how generators in co could use yield on arrays to wait on them all.

Also when do you need to use Promise.race?

<comment><USER3> await does "accept" arrays, it resolves to the array you pass it

<comment><USER2> It's just returning it as is right?

<comment><USER4> You can't await something that isn't a promise like an array. You should get a TypeError.</comment></comment></comment></comment>


PBS propaganda series about Vietnam - <AUTHOR> http://blogs.harvard.edu/philg/2017/11/11/pbs-propaganda-series-about-vietnam/ <comment><USER2> > _useful when [the US is] asking people to pay taxes at 2X the rates of the > most efficient countries_

This is uninformed to the point that it must be intentional. No one with even a passing knowledge of Western Europe could believe this.

"US taxes are low relative to those in other developed countries. In 2015, US taxes at all levels of government represented 26 percent of GDP, compared with an average of 34 percent of GDP for the 34 member countries of the Organisation for Economic Co-operation and Development (OECD)."

[http://www.taxpolicycenter.org/briefing-book/how-do-us- taxes...](http://www.taxpolicycenter.org/briefing-book/how-do-us-taxes- compare-internationally)</comment>