CloudFlare is ruining the internet for me (2016) - <AUTHOR> https://www.slashgeek.net/2016/05/17/cloudflare-is-ruining-the-internet-for-me/ <comment><USER2> > Not to mention their free tier doesn't cover the complex DDOS attacks that > you really should be concerned about

This wasn't true in 2016 and isn't true in 2019 either.

Whilst the system was named and blogged about in Sep 2017 [https://blog.cloudflare.com/meet-gatebot-a-bot-that- allows-u...](https://blog.cloudflare.com/meet-gatebot-a-bot-that-allows-us-to- sleep/) the internal repo has the first working code drop in Feb 2015.

The (D)DoS protection systems and team, which I am the Engineering Manager for, treat all attacks equally according to the nature of the attack rather than the target of the attack.

To address other parts of the blog though, there is nothing in our system that defaults to "do X for IP/Country y"... what there is, is learned state about traffic, clients, and the many layers of customer expressed configuration.

<comment><USER3> Yeah, that was my first thought when reading this.

It's not that Cloudflare is ruining the internet for you. It's the loads of illegitimate traffic and attacks that just so happen to originate in your country.

<comment><USER4> This parallels the conversation going on in the ML world right now.

"They just so happen to look like gorillas!"

This is not okay.

I know it's just a pile of linear algebra and there's no conspiracy to screw a particular country.

But once it's brought up that your system is causing harm by increasing inequality in the world, you can't just say "But that's what the algo is spitting out!".

At least _try_ to fix it.

<comment><USER3> > 1+1=2, but I want it to be 3, why are you not attempting to fix it?

How do you fix that which is not broken?

I understand the problem you are trying to point out, but the problem is not in the technology, it's in the application.

</comment></comment></comment></comment><comment><USER5> Companies like CloudFlare are contributing to the erosion of user privacy in the name of security. I experience the same kind of issues that the author of the article highlights, just because I take some privacy measures. It's my view that so-called security on the Internet, which implies confidentiality, integrity, and availability, is getting increasingly worse for end users. If I want to give up all of my privacy, the Internet is more convenient to use than ever. But for the privacy-conscious user, the Internet is becoming less usable by the day.

<comment><USER6> The article is complaining that CloudFlare (as a result of them refusing to track you) requires people to solve captchas over and over again. How is this hurting your privacy exactly?

<comment><USER7> Correct me if I'm wrong but isn't one of the strongest signals recaptcha uses to prove "I am not a robot" the fact that you're currently logged into a google account, have the associated cookies, etc? This "phoning home" from all these sites is a clear privacy challenge.

The good news is Safari and Firefox are making this harder and harder. The bad news is that you'll be solving a lot more captchas.

<comment><USER8> I wonder if privacy conscious browsers making their users have to solve more captchas will lead to more people disabling the privacy features... or there will be a time when captchas may become micro transaction based, so for a fee you might be able to skip them. A browser with support for an anonymous cryptocurrency to make the payments should be private and easy to use enough, but it might become an interesting world where you need to pay to remain anonymous.

<comment><USER9> Try creating a free email account on ProtonMail. I think they have an interesting approach very similar to what you're describing.

</comment></comment></comment></comment></comment><comment><USER10> Is the situation improved by using the Privacy Pass add-on [1] from Cloudflare?

Supposedly:

>Privacy Pass is a Chrome/Firefox browser extension to make browsing Cloudflare-protected websites a better experience for users. In particular, if a user IP address is designated to have a poor reputation then the user may have to solve a Cloudflare CAPTCHA page before they can gain access to such websites. Privacy Pass uses elliptic curve cryptography to generate 'anonymous' tokens after a single CAPTCHA page is solved. These tokens can be used in future engagements with Cloudflare websites to prevent having to solve more CAPTCHAs. The extension generates 30 tokens for each CAPTCHA solution and thus can be used to reduce CAPTCHA pages for each user by a similar factor.

[1] [https://support.cloudflare.com/hc/en- us/articles/11500199265...](https://support.cloudflare.com/hc/en- us/articles/115001992652-Can-I-use-Privacy-Pass-with-Cloudflare-)

<comment><USER11> So wait, "we shipped a bug, so we made a browser extension that lets you circumvent the bug". That's cloudflare's answer? I'm not impressed.

EDIT: people seem to be confused as to what bug cloudflare shipped. The bug is not having people solve captchas because their IP has a bad reputation. It's having them solve it _over and over again_.

You can put it however you want it, but if my app's UX is fine without cloudflare and it's shit with cloudflare, for a small but significant percentage of my users, then CF has a bug.

<comment><USER12> Thwarting denial of service attacks isn't a bug.

You seem to be confused about what your rights are around website availability. Hint: you have no rights. Absent specific coercion by government, the owner of the website had all the rights. If she wants to require you to solve a _Where's Waldo_ first, that's her prerogative. Your choice is to accept the terms or go elsewhere.

<comment><USER13> It's discrimination by country/region. It's like saying: oh, you are from Africa or Asia. The chance is higher you are a criminal, so do this test first.

<comment><USER14> Which is completely legal and encouraged. Here's an example: if you've ever shipped an ad-monetized free app, you've probably disabled regions like Russia, Iran, North Korea, etc.

You know why? Because the ad-revenue is worthless (and often malicious) and the users will be more trouble than they are worth. Same thing is happening with net traffic from other low value regions. One star reviews because users from $banned_region are complaining about lag due to their crappy wifi and/or some other issue you have no control over (defective ram in their 6 year old 2nd hand phone comes to mind)? Sign me up!

<comment><USER15> The hate for poor people in this comment is insane.

<comment><USER12> If insane is a new word for nonexistent

</comment></comment></comment></comment></comment></comment></comment><comment><USER16> The OP has posted a follow-up, but it's also from 2016: [https://www.slashgeek.net/2016/06/07/cloudflare-making- inter...](https://www.slashgeek.net/2016/06/07/cloudflare-making-internet- little-bit-faster-select-group-people/)

Two things that have changed since then are 1) Cloudflare's Privacy Pass browser extension and 2) their significant network expansion, both of which would be likely to affect the experience described.

One thing that has not changed, however, is how many (typically unsophisticated) web site operators actively search their logs for signs of suspicious / bot activity and then institute manual blocks in the hope of catching all of them. This is often done with very blunt instruments, such as whole-country blocks.

In contrast, people who are confident about their infrastructure can deal with the background noise of the Internet appropriately—by doing nothing.

<comment><USER17> I think a sizable portion of the small websites that use CF are on so much of a budget that they use a $10/month VPS for hosting then turn to CF for "ddos protection" in order to handle any spikes in traffic that would take down their under-provisioned server. Issue here is that CF's layer 7 flood protection (that's even on the free plans, you can use the rate limiting service for smaller-scale floods) doesn't take effect unless you are really getting hit hard, the cutoff is probably 500 requests per second or more.

When CF fails to protect against the small 30 request per second flood once, they're probably going to go through and add a bunch of aggressive blocks like blocking entire countries, as you've said.

</comment></comment><comment><USER13> Experienced the same during my Asia travels with Cloudflare 2016/2017\. Actually that was the biggest reason/negative ad not to consider Cloudflare when choosing a CDN. They simply don't respect Asia and other non first class countries with their standard settings. It's like good Internet only for the rich people in the world!

I'm aware on the Chrome extension (really, why should I install this sh&+t on the first place?) and that you can change Cloudflare settings. But the usual IT admin won't change this settings and f&+ck up Asia.

<comment><USER18> Asia is the #1 attacker of all websites I work with.

Since 100% of the traffic (based on our analysis) coming from Asia is not legitimate business traffic, how would you advise those responsible for these sites security to handle this?

Edit: I have no interest in using Cloudflare...

<comment><USER19> Similar for me. From the 732745 login attempts last month, 52% were from China, followed by US with 13%. Here is a graph: [https://i.imgur.com/YPAuTXO.png](https://i.imgur.com/YPAuTXO.png)

The sheer volume of bot traffic surprised me at first, especially since my website has zero human visitors as far as I can tell, but the numbers are consistent month after month.

Nevertheless, my $1/month VPS can handle the traffic without a problem, so I see no need to ban or rate limit any IPs, especially since I hate captchas with a passion.

<comment><USER20> > my $1/month VPS can handle the traffic without a problem, so I see no need > to ban or rate limit any IPs, especially since I hate captchas with a > passion.

I run a company that routinely scrapes government run, public domain websites. Sadly, many of these sites come with captchas. We can easily bypass these captchas by paying roughly $1.50/1000 captchas, but when scraping millions of pages a month, these costs become significant.

As far as I can tell, adding a captcha to a site does nothing to prevent bots, it just alters the economics of any business that relies on the data. I understand that bots can potentially slow down servers and cause disruptions for human users, but for the handful of government agencies that actually talk to us, we happily restrict scraping to certain hours of the day or limit overall traffic to a reasonable level. I would go further and happily give the money we're spending on solving captchas back to the government so they can upgrade their servers and make the system better for everyone.

For those that are conducting nefarious activities, captchas likely do nothing. For individuals, they are annoying. For legitimate scraping companies, they are a needless expense. Captchas are pretty obsolete.

<comment><USER9> > adding a captcha to a site does nothing to prevent bots, it just alters the > economics of any business that relies on the data

Definitely agreed. Recently I have been working on a side-project that makes use of bypassing/placating reCaptcha and it has been trivial and not so costly.

If it is accounts you're creating, it simply puts a "reasonable" price on account creation. If it is about scraped content, once again, does the same. However these costs already existed in terms of compute resources and time anyway. Captchas hardly made it any harder.

</comment></comment></comment></comment></comment><comment><USER21> Not to mention that it's not just a matter of filling some random words anymore but more like a "neverending" cycle of identifying buses and traffic lights which can last 5+ minutes. I must have identified more buses for Google than I've actually seen irl :(

<comment><USER22> Especially when you hop on a VPN; I kid you not I have to spent literally 20 straight minutes following this bs. It's infuriating for someone who self- respects their privacy.

<comment><USER17> Maybe try a personal VPN on digital ocean? You'd be guaranteed a clean IP that only you have access to, as it's very likely CF only flags VPN IPs because some malicious traffic is coming from them. The only issues with this are sites that block the digital ocean ASN (like Crunchyroll).

<comment><USER23> How many other VPNs are operating out of Digital Ocean?

<comment><USER17> I know torguard, at least for their wireguard endpoints, operates out of Digital Ocean.

<comment><USER23> Sorry, i was too vague. There are a bunch of VPNs running in DO. If you don't want to look like a VPN, it's not a great place to be.

<comment><USER22> Actually, not sure on DO's "relations" with VPNs. I've heard they're pretty strict on what you do since they're a somewhat small company that focuses on "droplets"/instances.

Personally I'd reccomend AWS or GCP given the free credits anyways.

</comment></comment></comment></comment></comment></comment></comment><comment><USER24> Please people do the internet a favor and disable the captcha. Go to Firewall > Settings > Security Level (captcha) And set it to "Essentially Off"

[https://support.cloudflare.com/hc/en- us/articles/200170096-H...](https://support.cloudflare.com/hc/en- us/articles/200170096-How-do-I-turn-off-the-Cloudflare-Captcha-challenge- page-)

</comment><comment><USER25> As someone living in Hong Kong I hate cloudflare. Way too many websites behind cloudflare block hong kong or cause me to have a captcha. This causes me to often route my traffic to my own vpn to counter that which is really annoying.

At least I'm lucky enough that my VPN is for some reason still not detected as a cloud hosting provider by either cloudflare or netflix.

<comment><USER18> What do you suggest to the people running websites where 90%+ of the hack attempts come from Hong Kong?

<comment><USER26> Please stop spreading this false narrative: [https://www.slashgeek.net/2016/06/07/cloudflare-making- inter...](https://www.slashgeek.net/2016/06/07/cloudflare-making-internet- little-bit-faster-select-group-people/)

Most network attacks and Spams actually comes from the United States.

<comment><USER27> I think they were suggesting the websites they run, 90%+ of the attacks come from Hong Kong, i.e. not that 90% of all website attacks come from HK. No way of knowing unless they publish the logs/stats

<comment><USER25> I think most likely they are amalgamating Hong Kong and China like some people do on the thread here. And yes, indeed some Data Centers in China are a big source of spam and automated attack, but it just doesn't make sense to block an entire country when you can just block hosting providers.

</comment></comment></comment></comment></comment><comment><USER28> > reCAPTCHA prompts happens a lot, I know its only one click away

This actually isn't true when browsing over Tor. It always makes you do 5 or more rounds of "click the traffic lights", "click the bicycles", etc. I don't know why. It seems to go far beyond checking that you're not a machine, and I suspect they're just abusing Tor users to get free machine-learning training.

<comment><USER29> Per CloudFlare themselves: "Based on data across the CloudFlare network, 94% of requests that we see across the Tor network are per se malicious."

[https://arstechnica.com/tech-policy/2016/03/new-data- suggest...](https://arstechnica.com/tech-policy/2016/03/new-data- suggests-94-percent-of-tor-traffic-is-malicious/)

[https://blog.cloudflare.com/the-trouble-with- tor/](https://blog.cloudflare.com/the-trouble-with-tor/)

<comment><USER28> A sensible CAPTCHA would be fine, but reCAPTCHA is borderline malicious towards Tor users.

</comment></comment></comment><comment><USER30> Heh. I just found out last week my hosting provider had put my websites on cloudflare for me.

They are not my hosting provider anymore.

How did I find out?

Traffic to the sites (all legitimate) fell circa 75% overnight. (non english sites)

<comment><USER31> > Traffic to the sites (all legitimate)

How do you know it was all legitimate?

I don't understand how cloudflare can stay in business if they cause a 75% drop in legitimate traffic to any website.

<comment><USER32> > I don't understand how cloudflare can stay in business[...]

VC backing and recent IPO have them swimming in cash, but they currently aren't profitable.

<comment><USER33> Out of all the recent S-1s they have one of the strongest aside from Zoom.

</comment></comment></comment></comment><comment><USER34> This.

I route all my traffic via mmy own vpn server at Hetzner for privacy and security reasons and this Cloudflare bullshit is infuriating at times.

Besides I guess 95% sites that use their free tier either don't actually need it or would be better off without it.

<comment><USER35> If you're in Europe and it happens on the website of a European company: make it a GDPR case. If you need to solve a captcha to access the privacy policy, they are clearly in violation.

<comment><USER36> Can you explain why this is a violation, please?

<comment><USER35> Privacy policies need to be immediately accessible to users. Hiding your privacy info page behind captchas, using unclear names for links ("service status" as a link to privacy info for example), making users click through multiple pages to find it etc makes you non-compliant.

Basically: you cannot hide the information, you cannot make users jump through hoops (captchas, require signup/login, pay for accessing) to read them.

<comment><USER36> Thanks, that's very useful.

Presumably the fact that it's not the site owner mandating the captcha, but an intermediary service provider doesn't matter then?

<comment><USER35> It really shouldn't, because the site owner is the one making the choice to use CF; CF is acting on their behalf (and the security-settings the site owner chooses at CF does influence whether and how often captchas are shown to users, i.e. "I'm under attack" mode). It would be different if the user's ISP did this.

This is another related issue, too, as CF is a data processor, so the controller (=site owner) needs to make users aware that their data is being shared with CloudFlare, as SSL terminates at CF, the content is analyzed and it's then (optionally re-encrypted) transmitted to the origin.

</comment></comment></comment></comment></comment></comment><comment><USER32> They have an extension that addresses this: [https://support.cloudflare.com/hc/en- us/articles/11500199265...](https://support.cloudflare.com/hc/en- us/articles/115001992652-Can-I-use-Privacy-Pass-with-Cloudflare-)

</comment><comment><USER37> As someone who works for a CDN (not CloudFlare), this was certainly interesting to read. It is always good to remember that we owe something to our customers' end users in addition to our direct customers, the website owners.

We have to remember that those are real people and not just percentages of traffic that are affected when we make decisions like putting a captcha up against every visitor from certain countries. I like to think we were already doing that, but it is good to be reminded.

However, I am not quite sure I am getting what the author is suggesting when they say that sites should forgo a CDN. Maybe I am biased, but if you thought latency was bad when a datacenter near you went down for maintenance, try having to go all the way to the sites origin in New Jersey for every request. I am not aware of any way besides a CDN (or a CDN like setup) that would get you good performance for people in all countries.

So I get the frustration with the captchas, and I get the frustration with the lack of multiple datacenters near you, but I wonder if you will make things worse for yourself by advocating to not have a CDN.

<comment><USER38> If you have a proper website without a gazzilion of useless JavaScript frameworks and custom fonts, there's hardly going to be an unacceptable experience latency-wise even if you have to go all over the world to fetch those resources.

<comment><USER37> I guess I didn't realize, until another commenter replied, that their traffic was actually blackholed... I thought it just had really bad latency, which they were complaining about.

</comment></comment></comment><comment><USER39> [2016].

Cloudflare's been working on improvements to the CAPTCHA system for Tor users ([https://www.zdnet.com/article/cloudflare-ends-captcha- challe...](https://www.zdnet.com/article/cloudflare-ends-captcha-challenges- for-tor-users/)), maybe some of those have benefited foreign countries as well. And CloudFlare does do country-level blacklists (which show up as the CAPTCHA behavior described) so maybe StackExchange had/has an overaggressive firewall.

</comment><comment><USER38> I don't know what's so unique about my connection, but I, too, receive these capchas on a daily basis, on the very same sites.

I'd rather they track me, by IP address and/or cookies, and stop this non- sense. I accept the cookies and have a static and dedicated IP, but there's never an end to Cloudflare's captchas.

<comment><USER40> Do you routinely use a VPN or a shared connection like University housing?

<comment><USER38> How do you define a VPN? Noone other than myself has ever used my IP addresses in X years.

<comment><USER41> Honestly I wonder if even an unusual setup like a static IP would register on one of their heuristics. It could flag as a web server, which would theoretically be more likely to ping a site rapidly than a personal computer.

<comment><USER42> > Honestly I wonder if even an unusual setup like a static IP

I have internet at home through Comcast, a huge and universally hated ISP in the USA. I also run a home server (though it only takes incoming connections, and does not visit websites itself). I purchased a dynamic DNS service that would update my domain's DNS whenever my home internet's public IP changed.

In over 5 years and through numerous modem reboots, my IP address has not changed once. A year ago I transferred my domain to another provider; I did not bother setting up dynamic DNS again and my website still works fine.

I have not purchased a static IP from Comcast. When I initially set up the server I had read that my home's IP address can change anytime the modem reboots, or possibly anytime at all, to any IP in Comcast's pool - which is why I subscribed to dynamic DNS.

So a static IP may not be as unusual of a setup as you say it is.

</comment></comment></comment></comment></comment><comment><USER43> What is the alternative solution here? How do you effectively protect access to websites while simultaneously making sure you never get false positives?

I really don't think going _back_ to not having Cloudflare-like services is a step forward. Is there another way?

<comment><USER13> It's afaik only Cloudflare's way. You don't see that behaviour from Google or Facebook. They really now how to handle the Internet in the "second" and "third world". Cloudflare is like: we don't care about you, you are too poor or a bot. Here is a extension if you are tech-savvy enough to find and install it.

<comment><USER44> From my perspective Google is far worse. I don't ever see Cloudflare's Captchas but I get Google's all the damn time, because I try to avoid Google properties. They intentionally punish people that aren't regularly using their products.

</comment></comment></comment><comment><USER45> Is it possible to have a "Cloudflare-like" service without using Javascript? I have Javascript turned off by default (no script) and run into the Cloudflare wall all of the freaking time, so much so, I refuse to use it on my websites.

<comment><USER46> You can set the security level to "essentially off" and the vast vast majority of your users shouldn't have to face the JS browser check. That's what I do on all of my sites.

The default is medium, and the sites where you run into checks all the time are probably on high.

Ref: [https://support.cloudflare.com/hc/en- us/articles/200170056-W...](https://support.cloudflare.com/hc/en- us/articles/200170056-What-does-Cloudflare-s-Security-Level-mean-)

<comment><USER47> Does anyone know why you can't turn off Cloudflare's security?

<comment><USER46> IIRC you can only turn it off completely on a business or enterprise plan. Probably a bandwidth-saving measure, or an upgrade incentive, or both.

</comment></comment></comment></comment><comment><USER48> There is some presumption that Cloudflare's behavior related to "second class" regions is based on statistics - percentage of traffic that is of ill-intent. Maybe that's not the case, but I suspect so.

CF customers probably know that any delays cost viewers/visitors, but losing a few good visitors is worth preventing a ton of bad visitors. And CF generally seems very thoughtful about their actions. If they make something unpleasant, they're likely to have a good reason for doing so.

<comment><USER38> Most properties protected by Cloudflare appear to be webpages that could easily be cached without any ill effects, for hours to days or weeks.

Why exactly do they need captcha protection?

<comment><USER49> That's like asking a gun shop why americans need so many guns

</comment></comment></comment><comment><USER50> These especially suck in China. It's especially painful when you are finally free and clear of the great firewall only to be stopped by a mistuned fraud filter.

<comment><USER17> Assuming the website itself isn't blocked by the GFW, many site owners regularly either set a Cloudflare rule to block or challenge (recaptcha) all china traffic (or even all non-US traffic for US-only businesses) due to the amount of spam and malicious requests that come from IPs originating in those countries. My own sites has shown some attacks from within the US but not nearly as much as from China before I set up a firewall rule to block China.

</comment></comment><comment><USER51> I live in the EU, but use a self-hosted VPN (for privacy reasons) on a German IP (digitalocean). And I too get the annoying reCaltcha prompts or even bans "your IP range has been blocked" all the time.

PS. To be fair it's not just coming from Cloudflare-protected sites. Webmasters and SaaS-app devs add this to their WAF layers everywhere. :(

<comment><USER52> It's probably less about the German IP and more about DigitalOcean.

Most people live in homes, not datacenters, which means that web sites expect human traffic to come from residential IPs, not datacenter IPs. What comes from the datacenter IPs is an endless stream of costly abuse, so they get CAPTCHA'd (if they want to support use cases like yours) or blocked (if they don't care about those)

</comment></comment><comment><USER49> Huh, I never had problems with cloudflare I used something like tor.

But I had TONS of New Zealand websites captcha me without a good reason. They all used some shitty local providers.

<comment><USER53> Where were you connecting from? From within NZ, unsurprisingly, I've never hit a CAPTCHA barrier on a NZ site.

</comment></comment><comment><USER54> I totally get it. For about 2 years I was subject to captchas for visiting websites of restaurants. No one else felt what I did because I don't have (never have and will never have) a Facebook account. What that meant for phone users trying to see the menu of 25% of any restaurant - a Facebook captcha. I think the only reason it went away is because I complained about it in hn. I think I got lucky.

</comment><comment><USER36> As a CTO of a startup trying to build a product on the web, though, CF is awesome. It solves a whole bunch of problems for free, and I don't have to worry about stuff that I would otherwise have to worry about (and I don't have time to worry about those things on top of all the other things I have to worry about).

In a few years, when I have a team of engineers and can spare the resources/expertise, we'll come off CF and do it properly. Until then, CF is a great service.

Sorry that your experience from SEA is not great, but tbh we're not selling in SEA, so any traffic from there is just a resource drain on our servers. Anything to discourage traffic from areas we're not serving is a positive for us.

Anything to discourage bot traffic is a huge positive. CF won't stop the bots, but costing them some minor amount of money per visit is still positive.

</comment><comment><USER55> I live in a developing country and I did experience an odd surge in recaptcha triggering for sites behind cloudflare around the end of 2016, but it really hasn't happened since.

It's barely anything at all, but my personal website is behind cloudflare and I've never had any trouble.

</comment><comment><USER56> Networking in Asia seems to be really complicated due to the various political interests in play. I remember our Cloudflare rep explaining that China had two or three telcos that refused to exchange traffic directly so you had to send everything through Japan and back to the telco you wanted in China.

I've always wondered how all the expats in Bali accomplish anything. Everybody says they can work remotely via the internet and run their businesses, but when I was there the internet would go down for the entire island frequently and could be down for minutes or hours.

</comment><comment><USER57> SSL is terminated on CF and website owners are allowing CF to generate SSL Cert and _Key_ for them in free plan. It is a wildcard certificate.

CF has the ability to read/alter the information we are sending to (or receiving from) the actual website. CF also has the ability (I do not mean they do) to impersonate the original website without the owner's knowledge, and with visitor's trust.

</comment><comment><USER58> In regards to the accidental censorship of content for users behind VPNs and proxies, on my site I have set the security level to essentially off and disabled any and all integrity checking of visitors. I haven't tested the effect on those users but it should allow anyone to retrieve the pages unmolested.

</comment><comment><USER59> The upside is that the 'second-class' markets will be served by local services. And when western markets are completely saturated, western site owners will be scratching their heads wondering how they get any part of the 'second-class' markets.

</comment><comment><USER60> Everything is collected. From the Boot. Bomb Utah.

</comment><comment><USER61> Hmmm...well unfortunately a lot of scammers and thieves on the internet come out of third world countries. Like, it's crazy, but the economics of scam artistry make this worth while as a career in certain countries - for example Romania ([https://abcnews.go.com/International/journey-hackerville- rom...](https://abcnews.go.com/International/journey-hackerville-romanian- city-reputation-criminal-hacker-breeding/story?id=60123285)).

So, websites are protecting themselves as best they can while pissing off the fewest of their customers by putting up more security measures blocked by country IP.

The real issue is the authoritarian governments that are making it so countries (like China) are completely fire-walled off from the internet.

Have you heard that Iraq is having massive street protests at the moment? If not, it might be because the government cut off internet access so a lot of the stories and media about the protests are not making it onto the social media sites. That's scary.

</comment><comment><USER62> How is it Google.com doesn't require captchas to open the home page? Nor does apple.com or any number of massive-traffic sites. The idea that captcha is the best, most effective, or most secure way to protect against DoS is just lazy.

<comment><USER63> To open the home page? True... But to view search results? I get captcha demands from google all the time depending on the search query (sometimes a query I make looks like something a bot might supply, like searching for a few IP addresses in a row) or if I'm using a VPN or Tor.

</comment></comment><comment><USER64> OP lives in Southeast Asia and suggests website owners choose a different CDN and fails to propose an alternative. ¯\\_(ツ)_/¯

There are two complaints,

(1) cloudflare requires a captcha for visitors from some regions (like SouthEast Asia)

(2) cloudflare does not have enough nodes in SouthEast Asia, and OP feels being rerouted to another node defeats the purpose of a CDN.

Yet, Cloudflare does (1) because they often see attacks from those regions. I'm not sure blaming Cloudflare for this is the right strategy. Regarding (2), CDNs do not just benefit users, they benefit the website too. Getting rid of the CDN is not a solution. Is there a better free CDN for that region? Is multi-CDN easy to setup?

<comment><USER38> > Getting rid of the CDN is not a solution.

Why not? The whole everyone-needs-Cloudflare is a made up problem, which depends on many false narratives.

And why do we as website visitors have never heard of Akamai, yet it's hard to find anyone who's never seen these captchas from Cloudflare and Incapsula?

> Is there a better free CDN for that region?

You can only find free mice in mousetraps.

<comment><USER64> > Why not? The whole everyone-needs-Cloudflare is a made up problem, which > depends on many false narratives.

I've never seen anyone say "everyone needs cloudflare" except maybe CF itself.

> why do we as website visitors have never heard of Akamai

Akamai is not free, they have a trial period that is free. It's not in the same space.

> You can only find free mice in mousetraps.

I don't understand your point. You feel CF is a trap?

<comment><USER38> > I don't understand your point. You feel CF is a trap?

Of course it is. If you aren't paying for service, you're not the customer, you're the product.

The whole mandatory TLS campaign is part of the lock-in, too.

</comment></comment></comment></comment><comment><USER65> Not to mention, CloudFlare has become synonymous with hosting and protecting the worst websites on the internet, with currently only two exceptions made, seemingly at the CEO's whim.

CloudFlare protects websites dedicated to doxing, stalking and harassing their victims. And when their victims complain, CloudFlare forwards all their personal information directly to said website owners while doing nothing about it. Websites that host content like the Christchurch massacre video and manifesto. Websites that have bullied people to the point of suicide.

I'm all for the importance of freedom of speech and being able to say offensive things on the internet, but CloudFlare is protecting sites that flagrantly violate the law.

<comment><USER17> The CEO's blog posts make it clear that CF really wants to bee the city water company that sells access to "dumb pipes": pipes that can route, filter and transform the stuff that goes through them while not knowing about exactly what it is that is going through them (at least when some maliciousness filters haven't triggered); the city water doesn't want to start cutting people's water off because of what they use the water for. Their stance is that they don't have the power to remove that content, all they can do is make the website vulnerable to ddos attacks and whatnot by kicking them off the service.

While 8chan being kicked off CF did bring them offline (their new anti-DDOS provider was told to kick 8chan off by the upstream bandwidth provider, and the domain hasn't worked since to my knowledge) the daily stormer is still working. CF kicking TDS off their network didn't stop their website from working.

<comment><USER65> I understand that, and I don't think websites should be deplatformed just for saying unpopular things.

But when a site clearly crosses into blatant criminality, it's disappointing when everyone who has the power to rein them in (Google with PageRank, CloudFlare with protection, the US government with criminal proceedings) decides to pass the buck on to someone else. It's always someone else's problem, meanwhile people's lives are being ruined and lost.

These sites would not be as highly profitable without CloudFlare's network, so I think it's fair that if someone wants to use CloudFlare, they're aware of what they're supporting when they give CloudFlare their money.

<comment><USER66> If it's "blatant criminality", sue them in court. Deplatforming is literally passing the buck to some other platform.</comment></comment></comment></comment>


A consumer jet airplane that takes off like a helicopter - <AUTHOR> http://www.samadaerospace.com/starling-jet <comment><USER2> > Refuel at fuel stations.

Can you imagine the gas station attendant's face when you pull up in this and ask for fuel?</comment>


Logging can be tricky - <AUTHOR> http://corner.squareup.com/2014/09/logging-can-be-tricky.html <comment><USER2> Reminds me of the time I sped up the main business app of a large company by 85% by removing "debug" logging. 2tb/hr of "made it here" isn't really useful at the end of the day.

Not the first time I've seen that by a long shot.

//shakes zimmerframe, shuffles off

<comment><USER3> Thus why your logging library should have a configurable log-level, and the first thing the user-facing log() function should do is check the arguments against the log-level and early-return if the message isn't important enough.

(In languages that support macros, you could also just make debug_log() a macro that gets compiled out on release builds, like assert(). That means a whole bunch of object files are going to be functionally different between debug and release builds, though, which is annoying for a lot of operational reasons.)

My favourite solution by far, though, is the one recently implemented by Elixir: log() is a regular function, but takes a closure instead of a string. That way, you can actually put all the work required to _generate_ the log message (inspecting data structures, doing intermediate lookups, etc.) inside the closure, and it _all_ gets elided when you decrease the log-level.

<comment><USER2> At least in Java, log levels don't buy you much when you wind up possibly rendering object graphs to strings (via your ohsohelpful custom, recursive tostring implementation), then duping it to concat "made it here" to it, then immediately abandon the mess on the heap for the gc to handle, since the production log level is warn, so your carefully built, highly detailed log message is never used. Repeat every other line or so... In tight loops, call iter.next in log message and on and on...

<comment><USER4> That's not a java problem, that's an "I don't understand Java logging" problem. The right way:

        log.debug("I created this gigantic object {}",
                  myBigUglyObject)

</comment></comment></comment></comment><comment><USER5> Grey text on white background? Please help!

<comment><USER6> You just dont understand sleek, minimal design. How pedestrian.

/s

</comment></comment><comment><USER7> I had a similar problem with java -- except that the entire application would freeze for double digit seconds. Another application would sometimes write a huge amount of data out very quickly to the fs cache. 30 seconds later (or w/e the expiration is), all those dirty bytes would get sync'd to disk more or less at once.

Turns out it was the JVM provided GC logging hanging on flush (not even fsync) calls. The flush call was during GC, and while the GC implementation held a stop the world lock. Digging through JVM source code is 'fun'.

<comment><USER8> I think I have exactly this problem. How do I confirm this? "strace" does not show me any obvious problems. Where would I look to see if the JVM "flush" was giving me problems?

<comment><USER7> Well, the easiest way to confirm is to turn off GC logging and see if your huge "GC pauses" go away. Alternatively, you could turn up the verbosity a bit - there's an extra flag for details about reference processing. In my case I was able to track down the exact line in the JVM from the log lines attributing the huge pauses to whatever trivial component occurred right after the first flush call.

If you record system metrics (eg. to ganglia) then you can also attempt to correlate large pauses to a large and rapidly declining number of dirty bytes in the fs cache.

</comment></comment></comment><comment><USER9> If all three machines had the same logging code, and one machine was fsync'ing slowly, isn't turning off fsync just a bandaid that hides the true problem?

When they discover the actual problem is some issue with the raid controller, I promise not to say "I told you so".

<comment><USER10> Hi, post author here. I didn't make it as clear as I could have, but the difference is that the problematic system had an unrelated process creating slightly more writes. I sort-of glossed over this with the 5%-20% difference in i/o util.

Unrelated write activity on a filesystem can cause cause fsync() calls in any other process to vary wildly in latency. This can be replicated, here's an experiment for you. First, run this:

strace -T -efsync ruby -e'loop { STDOUT.fsync; puts "a" * 120; sleep 0.1 } ' > ~/somefile

Then, in another terminal do a little bit of writing -- make sure it is on the same filesystem. For example:

dd if=/dev/zero of=~/someotherfile bs=4M count=1

On my poor little aws VM, here is what I see:

fsync(1) = 0 <0.025072>

fsync(1) = 0 <3.930661>

fsync(1) = 0 <0.024810>

That is, writing 4 megabytes in an unrelated process caused fsync() to jump two orders of magnitude.

Removing fsync() is an appropriate fix because we don't really ever want to flush this data to durable storage.

<comment><USER9> I probably read "an otherwise uniform cluster" too literally.

20% utilization on a resource that can only do 100 random operations per second is a major problem, and way different from 5%.

A piece of hardware that operates at 100 Hz, with 20% utilization, will block for about 135ms+ for the 95th percentile request. With 5% utilization, it will block for about 10ms for the 95% percentile request.

My quick calculations are somewhat below the 200ms discrepancy you show in the chart, but not far off.

Of course, turning off fsync is a perfectly good solution. Longer term, I would move to SSDs and just make this entire class of problem go away. I don't even have any spinning platters to test your strace on.

</comment></comment></comment><comment><USER11> While I really like the article and its analysis, I think there is a more general point here - scale is tricky.

Most programming abstractions and tools (such as Logging / UI design / persistence) across instances are all "easy" when it comes to one machine (or a few machines) in a closed, deterministic environment.

All of these become extremely tricky when you have to deal with many instances across many different possibly indeterministic environments. For UI, it quickly becomes what's the viewport? What's the real estate? How is interaction (touch? mouse? keyboard? numpad?). For persistence, one has to worry about number of simultaneous connections, character encoding and normalization, read / writes per second, synchronisation, etc..

So, also for logging - Are we logging client side or over the network? what's the memory on the client? What else is running on the client? What's the permissions through which we log? Do we care about a system call on this client given any user interaction? etc..

Logging, UI, Persistence, Network protocols, heck even just choosing an intermediate data structure are all tricky .. at scale across devices and indeterministic environments.

</comment><comment><USER12> Compressing the text logs prior to writing them to the disk also helps with these kinds of issues. You can also offload your logging to a dedicated thread and then use a lock free queue to increase your performance even more.

<comment><USER3> > You can also offload your logging to a dedicated thread and then use a lock > free queue to increase your performance even more.

Or use syslog() like a sane person, and let that "extra thread" live inside the OS IPC mechanism.

(Or stdout/stderr like a _modern_ sane person, and let upstart/systemd/docker/etc. push your logs to syslog if that's where it feels like pushing them.)

<comment><USER13> The OS IPC mechanism quite likely does not use a lock free queue -- or at least, not in quite the same way as I think the grandparent post refers to.

Using a well implemented ring buffer [+] can get enqueue operations down to a few instructions and something like two memory fences.

The overhead of IPC, which wakes up the kernel scheduler, switches the processor back and forth between privilege modes a few times on the way, knocks apart all the CPU cache and register state to swap in another process, while the MMU is flipping all your pages around because these two processes don't trust each other to write directly into their respective memory... is not going to have quite the same performance characteristics.

An moment in the history of logging is java's log4j framework, which, within a single process, used exclusive synchronization. When this was replaced by a (relatively) lock-free queue implementation, throughput increased by orders of magnitude. (Their notes and graphs on this can be found at [https://logging.apache.org/log4j/2.x/manual/async.html](https://logging.apache.org/log4j/2.x/manual/async.html) .) This isn't an exact metaphor for the difference between a good lockfree ringbuffer and IPC either, but it certainly has some similarities, and indeed ends with a specific shout-out to the power of avoiding "locks requiring kernel arbitration".

\--

[+] The "mechanical sympathy" / Disruptor folks have some great and accessible writeups on how they addressed the finer points of high performance shared memory message passing. [http://mechanitis.blogspot.com/2011/06/dissecting- disruptor-...](http://mechanitis.blogspot.com/2011/06/dissecting-disruptor- whats-so-special.html) is one of my favorite reads.

<comment><USER3> I think you're thinking of a synchronous RPC mechanism. I was talking about IPC mechanisms like unix domain sockets, where sending the message doesn't interact at all with the receiving process, but literally just sticks it into a buffer, where the other process can come and get it later.

</comment></comment></comment></comment><comment><USER14> Evan, I would be happy if you could explain more about what you see in the strace outputs. E.g.:

> Time spent in fsync accounts for almost exactly the delay we were seeing.

What delay? I see the whole thing taking 1.5 seconds and 1.3 seconds spent in futex (0.4 more than on the normal host). Not sure, why we are suddenly talking about fsync. I also don't know what either method (futex, fsync) could be doing.

All these are not questions I want answers to (some stuff I could google of course). I just want to show that it's a rather confusing read for some readers if you expect them to understand the strace outputs as well as you do, when you seem to be using that tool on a daily basis and the readers might not have used it at all, ever. It would be great to follow your insides better. Just small additions like the following would help a lot: "[The X seconds] spent in fsync [seen in diagram A] accounts for almost exactly the delay we were seeing [in diagram B]".

<comment><USER15> > Just small additions like the following would help a lot: "[The X seconds] > spent in fsync [seen in diagram A] accounts for almost exactly the delay we > were seeing [in diagram B]".

Exactly. I try to always point to the actual data when I'm telling my readers my conclusions.

"Calls to futex, epoll_wait and select are largely the same on both systems. The problem host does 1727 calls to fsync(), while the normal host does only 849 calls. The problem host spends 1.24% of its time in fsync(). The normal host only 0.48%. This makes fsync() our number 1 suspect."

This at least lets the reader refer back to the output and see corresponding entries.

</comment></comment><comment><USER16> Why would you fsync logs for a high-level service? Are you afraid a power outage is going to cause you to lose service logs?

<comment><USER17> Making sure your logging data gets written is relatively important.

Say, if you hit a bug somewhere that causes the server itself to crash.

<comment><USER18> How about installing a SIGSEGV, etc. handler to do the fsync (and perhaps even print a stacktrace)?

<comment><USER19> Disabling buffering in the file library should provide for a similar mitigation outlook. If the OS accepts the writes it WILL make it to disk unless the some part of the storage sub system(or the os itself) fails. So if the process dies recent writes should make it to disk regardless. But if the process triggers a complete system crash then you're kinda stuck needing fsync's :|

<comment><USER18> > Disabling buffering in the _file_ library should provide for a similar > mitigation outlook

Did you mean _logging_ library? I.e., application-level buffering? In that case, good point.

</comment></comment></comment></comment></comment><comment><USER20> Did anyone understand why logging calls on only problem machine had different fsync behavior than normal machines?

<comment><USER10> Hi, post author here. The slight difference in i/o (specifically: writes) was the trigger. I talked a little more about that here: [http://corner.squareup.com/2014/09/logging-can-be- tricky.htm...](http://corner.squareup.com/2014/09/logging-can-be- tricky.html#comment-1603001511)

And here: [https://news.ycombinator.com/item?id=8359556](https://news.ycombinator.com/item?id=8359556)

Even on the problematic host we only saw this latency issue in the 99th percentile. That is: even on the problem host 99 out of 100 queries were served as expected and only 1 out of 100 saw this additional latency.

<comment><USER21> Well, yeah, I noticed you guys' response to one of the comments on the blog post indicated that the problem machine had a different workload (additional tasks or something). That caused the additional writes, which then caused the latency for the main app on the box.

I think your point still stands about logging, being cautious about blocking I/O calls, etc. But, it seems the bigger point is one of how your overall system is architected, which proccesses run where, dedicating like nodes to their tasks vs. potential quality/consistency issues arising from having some pull double-duty, etc.

Those seemed to be the source of the real issue here.

<comment><USER10> Sort of. The catch is that even a very small write, say just a few megabytes, can drastically change the cost of an fsync(). On my test aws VM even writing just 4 megabytes one time is enough to trigger the problem. Even on an otherwise fully isolated system a few megs may be written from time to time, for example by a management agent like chef or puppet. Or by an application deploy copying out new binaries.

For example, here I reproduce the problem on a completely isolated machine: [https://news.ycombinator.com/item?id=8359556](https://news.ycombinator.com/item?id=8359556)

<comment><USER22> IMO the real issue is that a competent logging framework doesn't block app code to sync the log to disk. The buffer should be swapped out under lock, and then synced in a separate thread. Yuck.

<comment><USER23> The downside is of course that if you crash hard, the most valuable log entries are the ones least likely to be on-disk afterwards.

<comment><USER24> Which is why logging to disk on the server is BAD, have your log framework write to stdout and have upstart/systemd/whatever handle writing to a remote syslog server or whatever your fancy is.

</comment></comment></comment></comment></comment></comment></comment><comment><USER25> If you want tricky, try building a logging service and then enable logging to itself.

<comment><USER26> I once enable logging on an AWS S3 bucket... which was our 'logs' bucket... which was the stated destination for the bucket logging. It wasn't a particularly costly error, but there was a lot of logspam...

</comment></comment><comment><USER27> Does anyone think about doing logging to shared memory / memcached and then committing snapshots to disk at regular intervals via another process/machine?

If you're not all that concerned about consistency, each web server can keep their logs in a segregated memory space and then another process can combine/commit and send a flush command, leaving the primary machines relatively unencumbered.

<comment><USER10> With fsync() taken out of the equation this is essentially how a naive logger operates. The linux page cache plays the role of shared memory and only under rather heavy contention will a write() incur latency.

The trouble outlined in my blog post is that the logging framework was calling fsync() -- that is, specifically asking to flush the page cache all the way to disk.

<comment><USER28> Have you experimented with a log-structured filesystem? I once saw similar high-variation behaviour in a logging problem on extfs. Switching to NILFS pretty much got rid of variable latency entirely.

<comment><USER10> Not recently, no. I do know that a journaled filesystem can exacerbate this sort of problem as it can make extra work. For example: [http://lwn.net/Articles/328363/](http://lwn.net/Articles/328363/)

In a few cases in the past when dealing with unimportant data I have downgraded to ext2 for a nice performance bump.

<comment><USER28> Just to make sure we're talking about the same thing, a journalling FS isn't the same thing as a log-structured FS.

The first has a write-ahead log, the latter is basically just a log. So immediately writing to disk is relatively simple.

<comment><USER10> Yes. I haven't done anything with a log-structured FS. I have only played around with conventional filesystems.

</comment></comment></comment></comment></comment></comment><comment><USER29> Doe this increase the chance that you lose a bit of critical log if a fault causes the system to go down?

<comment><USER30> Yes, but if the logs are just for debugging an application, it usually doesn't matter. Application logs are useful for figuring out why an application crashed, but shouldn't be relevant for figuring out why a system went down. However, if the app crashes while some log information is still in the system buffers, the system will still write them out, even if fsync() is never called.

It's been a few years since I looked at this, but it seems like there's only a small window that data will stay in the buffer before it will get automatically flushed. The nice thing about this automatic flush is that it won't cause the application to block.

<comment><USER31> Also if your logging plugin / framework / module allows logging to remote syslog, that's a pretty good idea - no disk blocking. Although you are then risking data loss when the network goes down...

</comment></comment></comment><comment><USER32> Wow...this post made me think I shouldn't even attempt to run my own server infrastructure for my startup. This kind of analysis is way deeper than I'm currently capable of.

<comment><USER33> Hell, we're now infrastructure-less. No VM's (except for our QA to test on, doesn't count), no static servers we can't just destroy and spin up another of in <5 mins, and no database servers that we have to manage.

It truly is a joy.

<comment><USER34> Curious: How do you persist data with no database servers? Using an outside service to store it?

<comment><USER33> We used Azure SQL for our RDBMS requirements, and blob/table storage for our NoSQL stuff. Sure there's a server behind it, but it's transparent to us and it's geo-redundant and replicated. When i say "no servers" what I mean is, we don't need any system admins and very little ops folks. In fact at the moment, devs are pretty much ops since we need very little of it.

I also forgot to mention we have a build server, but thats just a virtual machine that we backup regularly. It's our only piece of static infrastructure. Everything else, except storage which is, again, transparent to us, is destroyed on a weekly, if not daily, basis for new releases.

</comment></comment></comment></comment><comment><USER35> I think this problem is mostly due to using Go, whose libraries are not mature.</comment>


One day, all news will look like this - <AUTHOR> http://www.qwghlm.co.uk/2010/06/07/one-day-all-news-will-look-like-this/

<comment><USER2> I can't figure out if this article is in support of it happening or not...

He's probably right this will start to creep in. I completely hate it. None of the videos are at all accurate or particularly ethical. They all tip right over in sensationalism and bad journalism.

Besides; we already use animations on western news - for legitimate things (like the oil leak). Dramatizing events we don't know about? Bad idea.

<comment><USER3> Artist renditions of news events have been published in newspapers for a very long time. I know this has been happening in Japan since the 1800's. I suspect this has been happening in the Anericas since the 1700's. What's new here? Is it the risk of photorealistic fabrications?

<comment><USER2> Obviously enriching reportage with graphics/animations is a good thing.

It's this specific sort that I dislike; badly researched and, essentially, fiction made for mass consumption (and feed our voyeuristic needs :)).

Don't get me wrong; if they did the proper research into all of those things and showed a realistic interpretation (the sinking boat, for me, was the closest to this) then good, that's enriching.

But I suspect it won't happen like that... :(

<comment><USER3> Probably the majority of the renditions in the Old Japanese papers were voyeuristic. It sells papers. Too bad supporting democracy throughh an informed public isn't as directly connected with profit.

</comment></comment></comment></comment><comment><USER4> This will unfortunately happen. I'm old enough to remember the first time "simulations" were used in news programs and there was an uproar over the ethics of doing that. Now, they don't even label something as "simulation" or "dramatization" or "re-creation." Thus news bleeds into entertainment which bleeds into news ... and Neil Postman spins in his grave.

</comment><comment><USER5> This is a very worrying trend. Consider this one from India - <http://www.youtube.com/watch?v=RtDb_vLZ0wc> I find it wrong on many levels. It is a drastic oversimplification of the events of the day and in some sense, implicitly portrays the pilot's job as something the newscaster could have done better.

News media has so far been vastly deficient in how they present technical and scientific matters and stuff like this will only exaggerate that problem.

<comment><USER6> For what it's worth, some newscasters present reports about certain events with a tone of voice as if they could have done it better themselves, without the aid of an animation. Sometimes it's hard to tell if newscasters are reporting the news, or chiding and scolding the individuals being reported on for human error.

</comment></comment><comment><USER7> This is one side of the coin -- it's getting easier to fabricate news.

The other side of the coin is that it's easier for people who spot fabrications to spread the word on popular blogs and such. When 60 minutes broke a story on what were supposed to be memos about a politician from the 1970s, bloggers who thought they looked suspiciously like they were typed in MS Word (and not on a typewriter) were able to link up with typography experts, do some solid analysis, and force CBS to issue a correction/apology. I've seen several bits of photoshopped "news" coverage exposed in recent years.

</comment><comment><USER8> The best one is the Conan/Leno story: <http://www.youtube.com/watch?v=-y04H5mFUFM>

<comment><USER9> Two thoughts.

a) Personally, I found that really informative, as well as entertaining. I didn't know the backstory of their feud.

b) I (still) have no idea if it's 100% true.

Yep, we'll probably see more of this.

</comment></comment><comment><USER10> "...as shown in this dramatic _imaginactment_." -- Stephen Colbert on the Daily Show several years back.

</comment><comment><USER11> Not as new as you'd think. CNET did this 15 years ago, when they were just starting up. They ran an animation of how O.J. Simpson supposedly committed the murders: [http://www2.iath.virginia.edu/spirit_trouts/mnookin/simpson....](http://www2.iath.virginia.edu/spirit_trouts/mnookin/simpson.html)

Incredibly tasteless. This is still the first thing that comes to mind when I see the name CNET, although I'm sure nobody else even remembers it.

<comment><USER12> Not as new as _you'd_ think, either: In 1918, animation pioneer Winsor McCay created the silent short film "The Sinking of the Lusitania", showing the RMS Lusitania being torpedoed by German U-boats. With remarkably realistic drawings, the film was presented as a documentary, but the visuals were imagined, based on verbal accounts and supposition. It functioned largely as a propaganda piece to increase anti-German sentiment during WWI.

The simulated sinking of the Korean ship in the article has nothing on this. :)

<http://en.wikipedia.org/wiki/The_Sinking_of_the_Lusitania>

[http://www.youtube.com/watch?gl=ES&hl=es&v=nN- KdPBhy...](http://www.youtube.com/watch?gl=ES&hl=es&v=nN-KdPBhyjc)

</comment></comment><comment><USER13> I did get a good laugh out of the Gordon Brown video. At around the 0:40 mark he punches some guy out in the corridor then throws a woman out of a chair and onto the floor, both accompanied by sound effects.

I'm not too worried by the idea that one day all news will look like this, though. These sort of animations are always going to be labour-intensive.

<comment><USER14> What is labour intensive now is going to be rather easy to do in twenty years.

Just think of some sort of Machinima based "gaming" program. Have your basic set of gestures for the actors, write up a little script, have someone control a camera, some voice-to-mouth animation.

Forget about twenty years from now, you could probably rattle off something in an hour or two with a modified game engine.

<comment><USER15> Or six years ago if you're willing to take some artistic liberties:

[http://www.thedailyshow.com/watch/thu-april-22-2004/crash- at...](http://www.thedailyshow.com/watch/thu-april-22-2004/crash-at-my-place)

</comment></comment></comment><comment><USER16> these look like some of the bumps that occasionally air on adult swim. I assume now that they are creating their own in the same style for laughs:

[http://www.youtube.com/watch?v=kI0QTaM- mo4&feature=chann...](http://www.youtube.com/watch?v=kI0QTaM- mo4&feature=channel)

</comment><comment><USER17> Somehow I can't see the BBC adopting this approach anytime soon...

</comment><comment><USER18> Inside Edition, Hard Copy, A Current Affair...

</comment><comment><USER19> The Running Man.

</comment><comment><USER20> lol, the brown video is actually well done</comment>


CISPA Is Ridiculously Hideous (And It Just Passed The House) - <AUTHOR> http://www.businessinsider.com/cispa-is-ridiculously-hideous-and-it-just-passed-the-house-2012-4 <comment><USER2> around the time of SOPA, at least 1 out of 10 posts was a post related to it and people were discussing about it. Now, it seems like people don't really care. I think they've won and we're now their little cyberbitches. What can I do, I'm Canadian. I'll just wait like a lot of people..</comment>


How to grow a saas company through content marketing - <AUTHOR> http://www.startupfrontier.com/2012/08/unbounce-growing-a-saas-company-with-content-marketing/ <comment><USER2> Unfortunately the linked article is a bit light on content. For one thing, I'd be interested in knowing how they built said network of twitter followers.

<comment><USER3> That's the thing about content marketing.. Of course it works! Duh.. if I had a Twitter follower base of 10K people, or an email list of 10K users, if I provide them something useful, they'll share it!

But all those content marketing experts fail to answer this: How do I get those people to listen to my content in the first place?!!

<comment><USER4> You've got your causality in the wrong direction. They have a Twitter base of 25k followers _because_ they provide something useful. Multiple times per day. With original content twice per week, on average, every single week for three years.

People choose to listen when you provide something useful, on a regular basis, in a channel that they're already paying attention to. Once you've gotten their attention, you can convince some of them to join a new channel for which you have better access, control, and tracking.

It's less about getting the listening than the continuous choice to proving something useful.

</comment></comment></comment><comment><USER5> I thing that content markting always works with good or poor content... But if you want really efective you need a team (or dedication) for this task.</comment>


Bears are very good Turks - <AUTHOR> http://riowang.blogspot.com/2017/05/bears-are-very-good-turks.html <comment><USER2> This is delightful. The sometimes unidiomatic translation lends it flavor. It's rare to find something so culturally rich on the internet, in English.

</comment><comment><USER3> That's really good gibberish. Unfortunately I see we are being spammed. How did this make front page?

<comment><USER4> Poemas del río Wang is far from spam! It's an extraordinary blog with a unique kind of whimsical erudition. Such posts will always be welcome here, as the site guidelines make clear: [https://news.ycombinator.com/newsguidelines.html](https://news.ycombinator.com/newsguidelines.html).

Please don't post snarky dismissals with no information in them. We're trying for a higher signal/noise ratio than that here. Not everything on HN's front page is to everyone's taste, nor should it be, and there is always much else to read.

<comment><USER3> My apologies dang, but it wasn't a snarky post. I've been seeing recent uptick in people posting spam links in comments, I assumed this was something akin to that. I'd assumed it was spam and was curious as to how it made it.

I did try reading it but as the other commentator mentioned it looked like it was indeed spam.

Mistakes happen, and I hope this site does indeed continue to keep the high quality comments going.</comment></comment></comment>


App-UI - <AUTHOR> http://triceam.github.com/app-UI/ <comment><USER2> Is it just me… or does this not look like it would "feel native" at all, from watching the video? (especially on iOS).

Not to deprecate the (I'm sure fast-maturing and ever-improving) work of the developer, but the lag and disconnect between its UI and the native UI just makes this completely _not_ "native-feeling".

<comment><USER3> Maybe because of this? <http://cubiq.org/remove-onclick-delay-on-webkit-for- iphone>

I didn't actually check the project source, so I could be wrong.

</comment></comment><comment><USER4> Some unsolicited advice.

"app-UI is a free & open source collection of reusable "application container" user interface components that may be helpful to web and mobile developers for creating interactive applications using HTML and JavaScript, especially those targeting mobile devices."

That's one loooooong sentence to start off your page with. Further, it seems like you're scared of diving into the mobile "niche." Embrace the niche. Try this, maybe?

"app-UI: Native-looking mobile application containers for HTML and JavaScript."

<comment><USER5> I disagree. The top sentence may be long, but it tells me exactly what it is and why I would want to use it.

</comment></comment><comment><USER6> This looks fantastic. However, after working on a simple HTML5 game for iOS, I'll definitely be going back to native for my next app.

<comment><USER7> Can you elaborate?

I am considering HTML for a mobile game, but I am concerned that performance will be lacking and CSS will be challenging.

<comment><USER8> I built the mobile HTML5 version of Words With Friends that launched as part of Facebook's failed mobile HTML app platform, aka "Project Spartan". (standard disclaimers: I don't work for Zynga any more, these are my own personal opinions, etc).

I much prefer web development to native iOS development for many many reasons, and am 100% for building desktop games in HTML5, but the thing about mobile web apps is that you'll spend every waking minute focusing on performance; you'd better be comfortable with not using jQuery, and you'd better be ready to get intimately familiar with the intricacies of how WebKit renders the DOM so you can fine-tune by hand the way it applies hardware acceleration. If you're lucky, with tons of optimization you _might_ get performance to a barely-acceptable level, and that's assuming your game's running in MobileSafari instead of a Nitro-less UIWebView. Things are only going to get better from here, but they've got a long way to go before I'll voluntarily write another web-based mobile game.

Also, if you're building a pure web app (as opposed to a native wrapper around a UIWebView), you functionally won't be able to have sound. I've heard that iOS6 is finally introducing a real web audio API, but the current version of iOS doesn't have a sane way to play audio in a way that's robust and full- featured enough for most games.

(And if you care about cross-platform support, don't get me started on the clusterfuck that is the Android browser. I'd honestly rather have to support IE 6/7/8 than the Android build of WebKit.)

<comment><USER9> totally agree. Android "Browser" is like the IE6 of mobile web. I bump into so many problems trying to tame android "Browser". Lack of hardware acceleration for simple displacement animation, the best transition animation you can apply is probably fade in and out not more than 2 stacked layers. The canvas was missing toDataURL method, canvas pixel manipulation with alpha channel resulted in some fuzzy pixels scramble. Weird virtual keyboard behavior. It takes few seconds for the device to tell the browser that orientation has changed, sometime has to be accompanied with light shaking of the device. No svg support for some older models. And majority of android devices out there are not running the latest Android OS. Luckily Chrome is much better browser than Android "Browser". Kudos to chrome team to make the mobile web on Android awesome again.

</comment></comment></comment></comment><comment><USER10> It's cool to see another project on the same level of abstraction as Sencha Touch (<http://www.sencha.com/products/touch/>)

</comment><comment><USER11> Great concept but the execution isn't as good as mobile competitors like JQuery Mobile which is pretty but incredibly simple for anyone with basic HTML skills to build something with.

<comment><USER12> This toolset has a _completely_ different use case from jQuery Mobile. jQM prescribes that your backend responds with typical HTML web pages. While it's conceptually possible to write an app that loads something like JSON into jQM, it's not straightforward.

This is more of a lower level transitions toolkit - only one piece of what jQM does, and I would imagine easier to fit into your hand rolled js app.

</comment></comment><comment><USER13> This looks really good. I just downloaded one of the real world example apps the page links to and it seems pretty awesome.

The scrolling isn't the smoothest but maybe I can find a fix somewhere between this and hnmobile.

</comment><comment><USER14> I've never seen jQuery described as a "solution accelerator framework" before.

</comment><comment><USER15> Very impressive. May try this out on some pet projects.

</comment><comment><USER16> Twitter Bootstrap for mobile app. Looks pretty good.

</comment><comment><USER17> Unfortunately it still feels broken on ios.

</comment><comment><USER18> Much needed components. Good job mate.

</comment><comment><USER19> Wonder why they depend on jquery and not zepto?

</comment><comment><USER20> You should release these as web components for better modularity and interoperability:

<http://tjholowaychuk.com/post/27984551477/components> <https://github.com/component/spec/wiki>

</comment><comment><USER21> Why does everyone emulate iOS appearance? I'm so tired of seeing it. It desperately needs a refresh.

<comment><USER22> Better question. Why are you in the gray? People need to stop down-voting things just because they disagree!

It's a solid question. jQuery mobile also emulates iOS appearance. I think it's like when people made old windows apps, people would try to make there apps look native to windows and use the same color scheme windows did.</comment></comment>


MediaGoblin: Double your campaign contributions for the next 10k - <AUTHOR> http://mediagoblin.org/news/10k-campaign-matching.html <comment><USER2> In case anyone was curious what MediaGoblin is here is a video with an overview and update on the project. It was given at a local Python user group meeting.

<http://www.pyvideo.org/video/1418/mediagoblin-update></comment>


Self-flying drone dips, darts and dives through trees at 30 mph - <AUTHOR> https://www.csail.mit.edu/drone_flies_through_forest_at_30_mph <comment><USER3> The video is too stylized for me. I would love for a more steady camera showcasing the drone avoiding obstacles as it flies around. The jump-cuts and music detract from the cool tech I want to learn about.

<comment><USER4> I just don't understand asking a human to run the camera drone.

Presumably there's more than one of these, have one follow and rely on the pace drone for positioning.

It was pretty hard to watch, but really watching sensing and localization applied in a 3D space is extremely cool.

<comment><USER5> maybe a camera drone would see the one in front as an obstacle and constantly try to turn away from it :)

</comment></comment></comment><comment><USER6> Is... is the video intended to be a parody of something? It's like they took two shots of a drone turning slightly to avoid a tree in the middle of a field, and decided to hit it with a nuclear bomb of "MAKE IT FEEL MORE EPIC" in editing.

<comment><USER7> Yeah... I was expecting a drone to be weaving through a forest of trees. Not flying towards a single tree and maneuvering around it.

Still cool, but that music and editing was over the top.

</comment></comment><comment><USER8> I guess I was expecting something more like this goshawk flying through a forest, but maybe it's too much to ask for :)

[https://www.youtube.com/watch?v=p-_RHRAzUHM&start=100](https://www.youtube.com/watch?v=p-_RHRAzUHM&start=100)

<comment><USER9> Kiwi birds too: [https://www.youtube.com/watch?v=sdUUx5FdySs](https://www.youtube.com/watch?v=sdUUx5FdySs)

(Caution: On topic, but saddest short animation ever made)

</comment></comment><comment><USER10> Fancy camera work, but the obstacle avoidance is not convincing. The drone basically passes by one tree, and that's mostly it. Compare that to any bird that lives its life in the brush. Human built flying machines are fast and can carry huge loads but compared to animals we still lag far behind in control and finesse.

<comment><USER11> So does the brain and AI. What, I'm off topic?

</comment></comment><comment><USER12> For a comparison, check out Intel's video from earlier this year: [https://www.youtube.com/watch?v=Gj-5RNdUz3I](https://www.youtube.com/watch?v=Gj-5RNdUz3I)

</comment><comment><USER13> This is a very disappointing movie, it shows the drone dodging the same tree (or just branch actually) over an over again, there is no indication of its speed, it dodges only one obstacle every time before a cut/edit is done. Is there a reason this was not tried in a forest (as the title would suggest)?

</comment><comment><USER14> The insight into only needing the differentials in the next 10 meters makes a load of sense and clearly worked.

Why is everyone hating on the video? I thought it was light-hearted and fun.

<comment><USER15> yeah, only i wished the insight would have been an algo that determines the necessary distance based on the drone's speed

</comment></comment><comment><USER16> Compare to humans:

[https://www.youtube.com/watch?v=1MBW8zoZUR4](https://www.youtube.com/watch?v=1MBW8zoZUR4)

<comment><USER17> That's a little misleading: the video has a "do you want to see the crashes" link:

[https://www.youtube.com/watch?v=QgrvrD_GOvg&feature=iv&src_v...](https://www.youtube.com/watch?v=QgrvrD_GOvg&feature=iv&src_vid=1MBW8zoZUR4&annotation_id=annotation_103248233)

<comment><USER18> I'm sure the autonomous drone had a bunch of crashes too.

Flying with VR must be quite amazing anyway.

</comment></comment></comment><comment><USER19> I would describe that as "Drone dodges, ducks, dips, dives and... dodges".

</comment><comment><USER20> The way the trees glow white in the plane view at 1:09 -- could they be using near infrared cameras or filters to help specifically with tree detection? Nowhere in the color video do the leaves look that bright. The paper doesn't seem to mention it though.

[http://groups.csail.mit.edu/robotics- center/public_papers/Ba...](http://groups.csail.mit.edu/robotics- center/public_papers/Barry14a.pdf)

<comment><USER21> It definitely looks like infrared to me. The paper mentions that they use a pair of Point Grey Firefly MV cameras, and the reference manual for those cameras[1] says that "in monochrome models, the IR filter is replaced with a transparent piece of glass."

[1]: [http://www.ptgrey.com/support/downloads/10116/](http://www.ptgrey.com/support/downloads/10116/)

</comment></comment><comment><USER22> Im reminded of the movie Real Genius. This is most applicable as an autonomous weapon. Unavoidable and its going to suck.

</comment><comment><USER23> in case you are wondering: [http://groups.csail.mit.edu/robotics- center/public_papers/Ba...](http://groups.csail.mit.edu/robotics- center/public_papers/Barry15a.pdf)

20 watts, running on two ODROID-U3

<comment><USER24> Now I've got no idea where the $1,700 comes from - I was assuming it was ~$100 worth of typical rc flying wing, with $1500+ worth of cameras and computers on board...

</comment></comment><comment><USER25> I wish I could be more excited about this but the most probable use for this technology is making it easier for drones to spy on people or for cruise missiles to be able to better fly into someone's window. :-(

</comment><comment><USER26> Very impressive. I wonder if this is the beginning of the end of FPV racing as a sport?

If you're unsure what FPV racing is, see [http://fpvracing.tv](http://fpvracing.tv)

</comment><comment><USER27> The sad thing about this is as the technology advances, the likely first movers on this tech will be for military purposes.

Do enjoy what appears to be VR piloting at 1.28 however. First great useage of it I have seen.

</comment><comment> tianlin-shi I wonder how well this generalizes to other environments, especially cities with dense blocks.

</comment><comment><USER28> I instantly thought about Interstellar beginning scene...

</comment><comment><USER29> could it patrol the city for crime?

</comment><comment><USER30> This does go a long way to explaining speeder bikes -- the human rider controls the gross movement, but the bike itself does obstacle avoidance. Otherwise, there's no way a regular stormtrooper could fly those through the thick forests of Endor.

<comment><USER31> In video games, there are a lot of clever tricks they use to make the player more likely to hit enemies. E.g. the cursor slows down when you get close to an enemy, and bullets bend towards them. See this video: [https://www.youtube.com/watch?v=UL1_ht1EdAU](https://www.youtube.com/watch?v=UL1_ht1EdAU)

I wonder if tricks like these would work on real life vehicles.

<comment><USER23> >In video games

in console games

<comment><USER32> console games are video games too

<comment><USER33> I think the distinction was between console games and PC games, which don't have auto-aim because the player is unlikely to be using a controller.

<comment><USER34> It's not a useful distinction since many kinds of games use player UX "cheats" like slight warping to physics algorithms, RNG weighting, etc. Most of these features act to take emphasis off of precise, low-level execution, or the pain of a string of unlucky results, in favor of higher-level strategy concerns, fast decision making, and expected values over time. There is an occasional chorus of "simulation purism" from some folks, but really, simulation elements are bulldozed in games constantly and consciously. The actual results are subtle and people don't take much notice when rules are bent, they just experience "fluid control" or "fair outcomes."

Auto-aim is only well-known because it's visually obvious when it occurs - <USER2> gun did not point in the direction of the camera.

</comment></comment></comment></comment></comment></comment><comment><USER35> The Navy really wants to get rid of pilots as it seems.

</comment><comment><USER36> > Barry's realization was that, at the fast speeds ... the world simply does > not change much ...

So is c the speed where nothing changes ? cold.</comment>


Ask HN: MacBook Longevity Tips - <AUTHOR> <comment><USER2> I don't know whether it's easy to do on a MacBook, but it's more and more common to limit the state of charge of the battery to for example 80%.

Today's batteries don't like to stay fully charged for long, nor empty.

<comment><AUTHOR> I've been using MacBook Pro 2019 on charger most of the time for around 6 months now. The BH is at 98% for me. I bought a hard case and keyboard cover, but ended up not using it as the case makes the display fall of a sudden while closing due to the additional weight. I also noticed any time I unplug and watch some YouTube without any background tasks,the fans start kicking in.</comment></comment>


YouTube.com/editor and Photo slideshows is going away - happy-go-lucky https://productforums.google.com/forum/#!msg/youtube/dx5xKn6v6rM/nvB59bbZBwAJ <comment> happy-go-lucky _This editor will go away on September 20, 2017. Remember to finish any unpublished projects before then._

[https://www.youtube.com/editor](https://www.youtube.com/editor)

</comment><comment> the-dude So now it is not being discontinued anymore, it is merely going away.

I wonder when it will return.

<comment> happy-go-lucky The forum post says _We've seen limited usage of these features, so we're retiring them to focus our efforts on building new tools and improving on other existing features._</comment></comment>


Ask HN: Web Devs: What do you use for project management and billing? - <AUTHOR> Curious to know what other web design and development teams are using to manage their and projects and bill clients?
Our team charge time at an hourly rate. We only estimate, never quote, so it's important to keep clients updated on where things are sitting and let them interact on tasks when testing and reviewing.
In the end, after trying a few of the popular apps, we decided to build our own. We've been using it and evolving it for 5 years and are just now launching a beta and trying to see if others might be interested
https://tasman.io/
https://vimeo.com/223398703
In my experience, all the features can be met by different apps but I haven't seen anything yet that handles the end-to-end workflow and has a nice user interface for users and end clients
What do you use? for client interaction -> project and budget tracking -> task tracking -> time entries -> billing <comment><USER2> I currently use Billings Pro, but thats just because I haven't had time to replace it yet.

Have/would you consider making it open source (no reason you can't still charge for your hosted version)? I'd love to use something that already exists, but OSS is a must for me for a new tool.

<comment><AUTHOR> Yes, Open Sourcing it is definitely something we're considering.

</comment></comment><comment><USER3> Trello for planning boards (free) and Harvest for billing by time (modest fee for several clients). I wish Harvest had a simple way to let clients check timesheets though.

</comment><comment><USER4> Liquid planner</comment>


The Myth of the Abrasive Genius - <AUTHOR> http://scienceblogs.com/principles/2009/06/the_myth_of_the_abrasive_geniu.php

<comment><USER2> I think the only example of the insightful outsider was the fellow who ~ 80 years ago who went around espousing the continental drift theory ( <http://en.wikipedia.org/wiki/Alfred_Wegener> ) though his ideas weren't necessarily a complete model, which may have been the crux of the issue

</comment><comment><USER3> Of course this guy is going to disagree with the abrasive genius idea, because he himself is one of the conscientious and agreeable types, as evidence by his dismissal of IQ as disagreeable.

I don't know why scientists think they are immune to all the social rules that govern every other aspect of life. One of those is agreeableness. People like to be agreed with - right or wrong. This is true in science too. Scientists are humans. They make mistakes. They are biased.

Look at the scientists who have disagreed with the idea that humans are responsible for and can stop Global Warming. Their voice has been repeatedly stamped out by the rest of the scientific community.

<comment><USER4> You're not a Scientist if you don't go by anything under than what you observes from systematic study/experiment or logically deduce.

Proper Scientists can oppose each other not difference in subjective bias but through different objective experimental data.

There are plenty of other so-called abrasive or odd Mathematicians/Scientists in history. My main areas of study are Mathematics and Computer Science so I name more in those areas (though not all listed are necessarily Mathematicians): Kurt Godel, Grigori Perelman, John Nash, Theodore Kaczynski, Nikola Tesla, ...

I could keep going but the bottom line is not the abrasive/"whatever you what to call or define them" genius is not all that odd, certainly not the majority I would think.

</comment></comment><comment><USER5> I'm glad someone else is really taking the original article to task. However, I will say there is one particular Monte Carlo Methods researcher, who shall remain nameless, that _definitely_ fits this mold!</comment>


Just launched a start up, animated greeting cards service without Flash - <AUTHOR>

Animated greeting cards service, without use of Flash delivering really cool multi-layered cards with 3 dimensional effects creating magic on the screen in moments! www.likeshareit.com <comment><USER2> Very nice..more javascript goodies. Amazing use of div layers and animation via jQuery.</comment>


Carp Language - <AUTHOR> https://github.com/carp-lang/Carp <comment><USER2> Challenging scope, hopefully they deliver. All my wishes with them.</comment>


Singly Launches "App Fabric" Platform For Developers - <AUTHOR> http://techcrunch.com/2012/12/10/singly-launches-app-fabric-platform-for-developers-to-speed-up-integrations-with-dozens-of-services/

<comment><USER2> I don't know why nohorse's comment is dead--Microsoft has had the AppFabric brand going for at least a couple of years now. I gave it a poke some time ago and don't remember much beyond a rather complicated distributed cache...

<comment><USER3> Its a hosting environment for SOA type applications. It gives distributed corelation, monitoring, persistent workflow and caching capability for WCF Services. Incredibly powerful and a beautiful piece of software but wholly misunderstood by the industry as a whole.

We use it for our back end on a couple of things.

Yes it is a trademark as well which is going to be interesting.

</comment></comment><comment><USER4> They way they've used videos on their homepage to tell the stories of their customers is great. It's unique for a dev tools company to take this angle, but I think it's very smart. You should be selling the end and the means.

</comment><comment><USER5> They did a bitchin' job on the website design. Absolutely nailed it, the deme page (<https://www.singly.com/demo>) literally made me say... "Wow".

</comment><comment><USER6> These guys have a fantastically simple-to-use platform. We've been at a number of hackathons with them and they always get killer adoption because it's something you can get up-and-running within a few hours.

<comment><USER7> Hours?! I want that to be minutes. Lemme know anything that could be smoother at email@example.com :)

</comment></comment><comment><USER8> Oh I didn't realize they weren't in open beta or release. They've been promoting it at hackathons in Seattle for several months now.

<comment><USER9> I recently did Angelhack SV with a friend, and knew Singly was going to be big because almost every single team with a login feature was using Singly.</comment></comment>