No More Deceptive Download Buttons - <AUTHOR> https://googleonlinesecurity.blogspot.com/2016/02/no-more-deceptive-download-buttons.html <comment><USER2> Perhaps now that Google has taken steps to block websites that display these ads, Google should take steps to stop accepting these ads onto their network in the first place. Most of the time when I see those DOWNLOAD/PLAY buttons, they're hosted on doubleclick.

<comment><USER3> I'm not at all against this move from Google - it is good sense. However, to play Devil's advocate, what are the odds this was pushed down by the MPAA/RIAA or similar? This policy more or less directly targets sites that offer free online streaming or torrent downloads of Movies/TV/Music. The sites that wind up with these deceptive ads are typically sites that provide copyrighted content to their users.

Again, this is not a bad move. But I'm curious about the true motivations. If I were the MPAA, and trying to shut down the revenue stream of sites offering free streaming and torrents, this would be one of the ways to do it. That, or Google is simply sick of receiving takedown notices - and this is one method to take these sites out of their listings before even receiving the DMCA.

<comment><USER4> In my experience most free online streaming and torrent downloading websites have very small hard-to-find download buttons with a lot of fake "Download Now!" ads. So this would actually make torrenting easier.

<comment><USER5> By showing a full screen alert ?

There are also a lot of free file hosting sites with tons of those fake download buttons. Good luck downloading from them now.

<comment><USER4> I mean it makes displaying "Download Now!" banners less effective. You can bypass the alert; Chrome puts the alert if I understood.

</comment></comment></comment></comment></comment><comment><USER6> This "Google Safe Browsing" initiative seriously worries me. It's effectively some unknown, mysterious, un-contactable set of AI algorithms/people/who knows what controlling the internet because Google owns everyone's browser.

One of my websites got tagged as "Dangerous" and having "harmful programs" despite having nothing of the sort. My guess is a silly hiccup of their neural network algorithms. And I have absolutely nobody I can contact about the issue to get an explanation. They just effectively killed the site in one fell swoop.

<comment><USER7> As with most things in this world, you could sue them.

<comment><USER8> They'd invoke the first amendment. It would be pretty open and shut.

<comment><USER7> I'm not sure if you know what the "first amendment" is. I'll tell you what it isn't, it's not a magical trump card that lets you say whatever you want.

<comment><USER8> [http://searchengineland.com/another-court-affirms-googles- fi...](http://searchengineland.com/another-court-affirms-googles-first- amendment-control-search-results-209034)

That seems to be how Google is (successfully) using it.

<comment><USER9> That's a VERY different situation. Google is claiming there that (essentially) they have the right to put sites in whatever order they want, and US courts are extremely sympathetic to that argument.

Here the issue is that Google is making direct, verbal claims about other sites. That's not to say Google couldn't come up with a strategy to win in court, but the strategy would have to differ markedly.

</comment></comment></comment></comment></comment></comment><comment><USER10> This is a joke right ? We run Adsense display ads on our site and have to spend significant time every day reviewing and blocking new ads which try to use these deceptive practices.

Since Google clearly has the tech to detect this they should be implementing it at source on the advertisers (malvertisers). Instead they are pushing this down to the publishers and hitting them with penalties.

It's a clever ploy in some ways - Google gets the revenue from the ads and also the kudos from Joe Public for "being on the side of the consumer".

<comment><USER11> Firstly, this will work on ad networks other than Google, so it''s more broad reaching than anything they could do just within AdSense. This is good.

Secondly, and arguably more importantly, the way to stop these adverts is for them to cost the advertiser (in either money or time) without giving them the reward of revenue. If the ads stop working then people won't have a reason to make them. By stopping the ads in AdSense rogue advertisers would just change to a different ad network. The problem wouldn't stop.

This is a good move by Google.

<comment><USER12> "more broad reaching than anything they could do just within AdSense"

... as long as you don't care about browsers which don't run Google's Safe Browsing service.

You know another way to stop these ads? Make available an advertising network which doesn't serve them. Website owners who don't want to install malware on their users' computers - which is probably most of us - would prefer that network to the others. As-is, with even Google's network serving up malicious ads, the choice for a website that wants to run display ads appears to be either build out a sales team & manage inventory itself, or accept that some percentage of its users will get scammed.

<comment><USER13> I've actually built an advertising network[1] that is not focused on serving display ads, but linking to content directly in images. Video demo: [https://www.youtube.com/watch?v=8GfKBvs53Ss](https://www.youtube.com/watch?v=8GfKBvs53Ss)

The thought is that if "advertising" is actually a _feature_ of a website, then it solves the problem of users trying to avoid being shown ads. If you could hover your mouse over an object on any image on the internet and be taken directly to where you can buy that without all the hassle, I'd see that as a big win.

Note: Just onboarded our first customer yesterday. He's using it to promote iPhone cases based on his instagram feed[2]. Hover over the cases on a desktop, and you'll see what the case is. Click on it, and it takes you directly to the product page.

[1] [http://pleenq.com](http://pleenq.com)

[2] [http://www.obeythekorean.net/pages/instagram- feed](http://www.obeythekorean.net/pages/instagram-feed)

<comment><USER14> This is the first ad platform I've seen that is innovative in a good way, instead of the usual remarketing/tracking/native/data whatever bullshit. Seriously, awesome idea and execution. Have you gotten any press for this?

<comment><USER13> No press, just a reddit post[1]. Do you know anyone? :)

[1] [https://www.reddit.com/r/Entrepreneur/comments/43qnen/if_you...](https://www.reddit.com/r/Entrepreneur/comments/43qnen/if_you_own_a_website_i_want_to_make_you_more_money/)

<comment><USER14> This is undoubtedly interesting content for writers and bloggers. I'll think on it and PM you if I come up with anything worthwhile.

</comment></comment></comment></comment></comment></comment></comment><comment><USER15> So safe browsing is now blocking pages where the adverts are "often not distinguishable from the rest of the page"?

IMO that covers all of Google's search result pages. Most users don't seem to realise that the top results are paid advertising...

<comment><USER16> Here's a google SERP for "insurance": [http://imgur.com/VUqwyPq](http://imgur.com/VUqwyPq)

Are you saying the bright "ad" lozenge next to the paid results isn't explicit enough?

And anyways, users don't go out of their way to avoid clicking on ads unless the ads are utter crap. Google's whole business is to make the ads relevant to the search and the user, so who's to say the ad isn't actually a relevant result?

<comment><USER17> Somewhat off topic... but also, it states 1,010,000,000 results on the first page of this query but if I go to the last page, it comes down to 412 results... I can't believe that there is only 412 insurance related pages on the whole Internet... [https://www.google.com/search?q=insurance&num=100&safe=off&s...](https://www.google.com/search?q=insurance&num=100&safe=off&start=400)

<comment><USER16> I think there's some weird pagination happening in your URL. If I take the "num=100&safe=off&start=400" off, it works as expected.

<comment><USER17> It works as expected according to them if they don't expect you to look at all the results, I guess.

Please tell me what link you have at result #500, because I can't see it even if I remove the URL parameters and use the standard 10 results per page...

<comment><USER16> That gets me "Portal: Health insurance - University of Bern", on page 52. The url looks like:

[https://www.google.com/search?q=insurance&biw=1440&bih=830&e...](https://www.google.com/search?q=insurance&biw=1440&bih=830&ei=Yg61VsmWF8eZmQHF5aawAg&start=500&sa=N)

<comment><USER17> I cannot go past page 53 when using the URL that you are referring to (they added some results that can be seen, it currently is 532 but it changes pretty often).

But the behavior changed since yesterday... yesterday it would have said "Page 53 of 532 results" and now it says "Page 53 of about 1,010,000,000 results" but most of them still can't be seen...

</comment></comment></comment></comment></comment></comment></comment><comment><USER18> I'm surprised of the negativity towards this action, but I guess I shouldn't be. A lot of you are in a very different situation than me and this will affect you directly. However, warning people away from being potentially tricked by these deceptive ads is a very good thing.

Tons of sites out there that turn a blind eye to such ads and that's bad. Yes, there will be some unfortunate pain for sites that responsibly attempt to block these ads as they come up. Assuming the blog post is correct and Google has implemented this correctly, it should be minimal for those sorts of folks since they claim the penalty will only occur if users are _consistently_ getting social engineering ads. (I suspect Google will ratchet up the rate over time though, assuming these ads become less common as a result of this and similar efforts).

EDIT: See below. I'm convinced and with y'all now. Google--this is a step in the right direction and I support this action, but you do need to get your own house in order too!

<comment><USER19> > I'm surprised of the negativity towards this action

Its pretty basic. A lot of people are not happy that Google is serving these ads that they are telling you they will stop at the browser-level with a warning that makes the site owner look guilty. They want Google's Ad network to stop serving the ads in the first place.

<comment><USER18> Thanks for that.

So can anyone point me to an ad served by Google that is as bad as the examples in the blog post? I thought Google had previously cracked down on such ads from the serving side too, although less deceptive ones were still allowed, no? I'm happy to be better informed!

<comment><USER20> getpaint.net is mentioned elsewhere itt. Perhaps not quite as bad, but still at least mildly deceptive.

<comment><USER18> Gotcha. Yeah, I see one there. Big green button with a down arrow saying "GET IT NOW" lots of whitespace below and then a smallish rewaterpressure logo. Because of the whitespace, the button very much looked associated with the paint.net download text above, not with the rewaterpressure logo below.

I also received the less bad (but still bad) text-based "start download now" one the same page. You convinced me. Editing my parent post above.

EDIT: for those curious, here's what the page looked like on the page load mentioned above: [http://imgur.com/SisOXNT](http://imgur.com/SisOXNT)

They were all Google served ads.

</comment></comment></comment></comment></comment><comment><USER21> "You may have encountered social engineering in a deceptive download button, or an image ad that falsely claims your system is out of date."

Weeeellll...I see 99% of these in Android in-app ads. So - a) this should be the end of it, or b) someone is being a bit hypocritical here. I sure hope for the former.

</comment><comment><USER22> This is really important. One step closer to killing ad tech companies who only make money off my grandma and little brother.

<comment><USER23> You mean Google? I see a tonne of deceptive advertising propagated by Google's ad network.

<comment><USER22> No. I mean the companies whose sole business is to get you to download bundled malware, change your search engine, push ads to every site you browse using extensions etc.

Source: I've worked at one of those.

<comment><USER23> Surely the companies that advertise via Google to try and deceive you to into clicking their download button or whatever fit the bill? And surely Google, by actively enabling them, is also part of the process. As stated elsewhere on this thread, if they can detect deceptive sites, they can detect deceptive adverts.

<comment><USER24> So they implement detection, and remove these ads. The scummy advertisers then permutate their ads until they get past the detection and the game continues.

We're getting pretty good at image classification, but I don't think that extends to maliciously crafted inputs.

<comment><USER23> That or they manually verify each ad submission. If that violates their business model of high-volume low-value automated processes (and it obviously does) then you have to take account of that when decide how you view company. Automation and the inability to verify at the scale that they operate doesn't somehow absolve them.

<comment><USER25> Yeah, basically Google is looking for any way to avoid actually reviewing the ads they broadcast. Probably because advertising becomes drastically less profitable for them if they do. I feel when this sort of conflict of interest is occurring, where it's profitable for Google to continue shipping malware to users, they should be held legally accountable for their failure to police what they distribute.

</comment></comment></comment></comment></comment></comment></comment><comment><USER26> Interesting that when I read "these buttons", I thought " which buttons?" because my brain's spam filter initially made them invisible to me!

</comment><comment><USER5> I hope they'll include a "Stop the nanny" flag in chrome://flags as well.

I mean you can't even change the new tab page to a custom .html, without Chrome nagging you at every launch if the settings are correct. If you make a manifest.json and load as an unpacked extension, it will moan about that.

FFS, I know what you're trying to do with Joe/Jane Noob, but at least give me something to skip that if I know what I'm doing.

[edit] This wouldn't be even needed if new tab page was customizable. Now it it's just a Google billboard.

<comment><USER27> The hard part here is that, _wherever_ you'd decide to persist a "don't bug me any more about this" flag, malware could also potentially write that same flag to that same place. For example, Windows UAC is frequently set to the "don't bug me about this, just auto-elevate" setting by malware.

<comment><USER28> > The hard part here is that, wherever you'd decide to persist a "don't bug me > any more about this" flag,

Build a new binary - offer users to install a "developer" build of Chrome which is exactly the same as the mainstream "release" build, except it allows disabling the protections.

<comment><USER27> You don't actually have to go that far; there are plenty of Chrome settings controlled by command-line options, and that's usually safe enough—it's actually really hard for malware to "sneak in" command-line options (if the user is a regular user, while the the Chrome shortcuts in the Start Menu _et al_ were installed under elevation, which is the usual case.) There's a command-line option to Chrome that entirely disables the sandboxing protections, for instance.

My distinction was just that there's absolutely no way to have a _UI-based_ mechanism for disabling nags, since behind any UI is a persisted flag. If you're up for editing your shortcuts to add command-line options, that's fine.

<comment><USER28> > (if the user is a regular user, while the the Chrome shortcuts in the Start > Menu et al were installed under elevation, which is the usual case.)

Nope. Windows allows deletion of "protected" shortcuts e.g. from your desktop and launch bar.

<comment><USER27> You mean shortcuts placed in the All Users Desktop/QuickLaunch/StartMenu folders? (I'm guessing it's just "hiding" them with a Desktop.ini entry, rather than truly deleting them?)

That's probably fine, actually, as long as the user (i.e. malware) isn't allowed to create their own shortcuts to replace the deleted ones. I assume there's a GPO to disable the per-user Desktop/QuickLaunch/StartMenu folders, so that only the results from the All Users ones show up?

</comment></comment></comment></comment></comment></comment><comment><USER28> Sourceforge, download.cnet.com and friends won't like this... also many of the streaming/OCH sites will be affected.

<comment><USER29> Imagine the fit the various torrent sites will throw about it.

</comment></comment><comment><USER30> This is really awesome. I was so excited I actually clicked the fake download buttons to try to install it. So when does this role out?

<comment><USER31> I'm so glad that I'm not the only one! haha

</comment></comment><comment><USER32> This blog post is not talking about those Adsense Download buttons, is it?

</comment><comment><USER33> I wonder if those ads that push you to install certain product with half- truths and lies are considered deceptive as well. You know, "Install Chrome for better experience".

</comment><comment><USER34> Sounds good to me.

I'd like to see Google take this one step further: author and publisher (as in advertising network) accountability.

[https://plus.google.com/104092656004159577193/posts/Fitbdvdk...](https://plus.google.com/104092656004159577193/posts/Fitbdvdkw32)

Advertising providers, and advertising _publishers_, who forward "bad ads" are given a time-out.

Perhaps 10 minutes for the first instance, but increasing durations for repeat ocurrences. Days, weeks, and months for repeated gratuitious violations.

Ad providers and publishers who find they're being timed out for violating standards are, likely, going to clean up their acts, and find ways to ensure that mistakes _don't_ happen. Including direct vetting of content.

If Google don't block ads, I will.

Oh, wait, I already do that. But the rest of the Net is still catching up.

</comment><comment><USER35> Hmmm, I just went to google.com and typed "Chrome for Windows" Of the 10 links that appeared, at least 4 or 5 were for malware infected versions of Chrome. I followed the links, the download buttons on those pages are still shown. Maybe I misunderstood

</comment><comment><USER36> Does that mean they're taking down SourceForge?

<comment><USER37> It just got sold to BIZX, which I think promised to stop some of the practices.

</comment></comment><comment><USER38> Let's put Google's Youtube behind a deceptive site screen immediately! [http://i.imgur.com/cXf5dwJ.png](http://i.imgur.com/cXf5dwJ.png)

<comment><USER25> Yup, here's another: [https://plus.google.com/+JakeWeisz/posts/6YMTfeRw2jY](https://plus.google.com/+JakeWeisz/posts/6YMTfeRw2jY)

</comment></comment><comment><USER39> My AdWords account was suspended a few days ago because of, I presume at least, this: [https://www.rackforms.com/rackforms-express-for- wordpress.ph...](https://www.rackforms.com/rackforms-express-for- wordpress.php)

A popup in AdWords says I've violated "Unsupported content free desktop software". RackForms Express is a free version of my flagship product, that one actually being advertised.

You're feelings may very well differ, but if I've clicked on a link, organic or ad, and I get offered a solution to the problem I was seeking an answer for, that seem like a pretty good deal. If nothing else, this differentiation may well be the difference between getting a sale or not.

The appeal process is to fill out this form: [https://support.google.com/adwordspolicy/contact/advertise_s...](https://support.google.com/adwordspolicy/contact/advertise_software_pol), but from my reading in Google's product forms this doesn't always work. If this fails, the end result will be to remove free software from the net.

We're all for blocking those horrible "Download Firefox" ads, but this change, and the resulting aftermath, feels...I don't know, dangerous. If the end result is small companies like mine pulling valuable resources from the web, I think we all loose.

Fingers crossed my appeal goes through.

</comment><comment><USER40> Also, ideally Google should implement this within their search algorithm itself, by punishing the sites which indulge in such practices by pushing their search results much further away.

</comment><comment><USER41> So if a site is using Adsense and doesn't require all ads are reviewed then as site owners appears we run the risk of another part of Google telling visitors that the site is malicious. Hope that the site owner tooling referred to is good enough that it can identify the advert, at worst the network otherwise only option would be to remove Adsense code.

<comment><USER32> I tried to manually review all ads. But my experience was that this is not possible with the given tools. It would take hours per day.

</comment></comment><comment><USER42> I will take programs like this seriously when Google stops bundling Chrome installer with unrelated software like Adobe Flash.

<comment><USER30> Flash is unrelated to a web browser? I mean I hate flash as much as the next guy... but how can Flash not be related to a browser?

<comment><USER42> You want to install Flash for Firefox, it will install Chrome by default.

<comment><USER43> I did not install flash in FF, and I do not miss a thing. Sometimes there is this warning that not all content could be displayed, but I would not have a clue what functionality is missing.

</comment></comment></comment></comment><comment><USER44> Here are some more proactive approaches that would support their desired perception of caring about the browsing safety of their users:

* How about updating the radio buttons that appear when you "report this ad" to include "deceptive" as the reason for the report

* DoubleClick (by Google) serves the majority of these "Download now" ads that I've seen on sites that cater to the general public. Don't let advertisers run these ads. Do. Not. Allow. Many of them have "start download" in the plain text of the ad unit, and others are easily found by a bit of OCR on an image ad unit.

Please do not give me a Google-branded poncho (telling me how amazing it is with an infomercial about its revolutionary Dry Living Experience™) when you could patch the leaky roof to truly create a Dry Living Experience.

</comment><comment><USER45> The first thing they should do is harshly penalize websites with a lot of ads in their search results.

</comment><comment><USER46> I hope they block Sourceforge and their deceptive "Download" buttons as well.

</comment><comment><USER47> I tend to enjoy these type of announcements and discussions regarding Google, because it seems to remind the general population that while yes, Alphabet is a diversified technology powerhouse, at its core, its most primal competency is that of an advertising firm. A very successful advertising firm. Sort of like how Jerry Jones was a very successful oil business tycoon before diversifying his interests by way of purchasing the Dallas Cowboys and 'business-ing' it up to a multi-billion dollar brand. No oil, no Cowboys for Jerry. No ads, no self-driving cars for Google.

</comment><comment><USER48> Awe man guess I can't use chrome to browse The Pirate Bay anymore...

<comment><USER24> > Awe man guess I can't use chrome to browse The Pirate Bay anymore...

More likely, TPB stops running these particular shady ads, because chrome is such a popular browser.

</comment></comment><comment><USER49> Too late google. I already intsalled ad blocker on every computer in the house years ago because my parents would accidentally click "play" or "download" in deceptive ads.

</comment><comment><USER50> So who decides what to flag? Is google analyzing the behavior of chrome users and then automatically flags websites? Or can users flag websites? I disabled "Automatically report details of possible security incidents to Google" and "Protect you and your device from dangerous sites" in the settings, will my chrome browser still report these websites (in case it ever did)?

</comment><comment><USER51> I don't see the difference between "Buy this Pill to lose weight" and "Download this codec".

</comment><comment><USER52> "Pretend to act, or look and feel, like a trusted entity — like your own device or browser, or the website itself."

This is a broad statement. Taken at face value, this covers all native advertising - with articles/images/videos/thumbnails/etc intended to fit in with the content of the site/app.

<comment><USER53> If we can get rid of native advertising too, I'm all for it. Native advertising is a special kind of evil.

</comment></comment><comment><USER54> It's a laudable initiative to protect average Joe from himself, but I don't feel like Google (or any other company) deciding for _me_ what is dangerous. They should at least provide this feature as an opt-out option. Still, better option would be to educate more people of ad and script blockers.

<comment><USER55> [https://support.google.com/chrome/answer/99020?hl=en](https://support.google.com/chrome/answer/99020?hl=en)

There's been a setting for this stuff forever.

</comment></comment><comment><USER56> I hate using free hosted download services that some people use, especially on forums, because there are ads with download buttons and real download buttons and it's nearly impossible to tell which button needs to be clicked to initiate a download.

</comment><comment><USER57> Despite all the legal attacks in the UK and elsewhere, this is probably the biggest blow to piracy sites so far. When one gets taken down another spins up, but if you turn your adblocker off they all have these awful adverts.

<comment><USER58> For organic traffic pirate sites, the biggest blow was Google's algorithm change. For websites which used a freemium model, the biggest blow was PayPal, Visa and Mastercard banning file sharing websites from using them. Todays announcement was merely a nail in the coffin for those sites.

</comment></comment><comment><USER59> Welp, hopefully they will make it optional and we can turn it off.

</comment><comment><USER60> Amazing idea. It will save so many people form mal-links but even sophisticated users from trying to figure out which is the real "download" button

</comment><comment><USER61> I know these fake ads all suck and everyone hates them but somehow getting rid of them feels like cutting off a little piece of what makes the web the web.

I kinda like this darker, more free-for-all, wild wild west side of the Internet.

<comment><USER62> Me too man, I remember the internet back then where everything was unique and not a cookie-cutter bootstrap boilterplate. Want a nostalgia trip? Download Opera (one of the early versions) - it'll pluck at your heart strings and make you yearn for times when you had more personal responsibility and Google wasn't there to infect everything with it's nanny browser. Hell, even Firefox the last bastion of Freedom on the web, is following Chrome.

<comment><USER63> Mostly because there's more benefit to be gained in making the web usable by non-experts than in preserving the current status quo.

There's a reason the Wild West didn't stay wild.

<comment><USER62> I agree, but it still sucks that it seems you're being sold something at every turn. Websites are no longer there, just to be there. It's always about the upsell or agenda.

<comment><USER64> That's one reason why I'm a little annoyed personal websites went mostly the way of the Dodo and you can only expect friends to check things out if you give them direct links to some trusted site from a site they use all the time, like a link to a Youtube video from Facebook, or your Medium or Tumblr entry from Twitter.

I still like designing personal websites, but it seems like a waste of time now.

</comment></comment></comment></comment></comment><comment><USER65> Google should detect and block the buttons, not display full-screen warnings.

What next? "This site contains controversial views"? Or "Politically incorrect website ahead"?

<comment><USER66> Google can't block elements on sites it does not control.

Warning users of phishing and warning users of speech that it finds objectionable are 2 different things entirely.

<comment><USER65> A deceptive button does not equal phishing. It just might (and most often does) open a non-malicious popup with some ad. (non-malicious in a sense it won't install ransomware to your PC)

And of course Google can easily integrate it's own ad blocker in Chrome if it chooses so.

</comment></comment></comment><comment><USER67> More censorship... no thanks.

</comment><comment><USER68> This is a great move. Computers are extremely difficult and deceptive for those who can't remember the myriad of rules and gotchas.

What we really need is an open pledge of non-deceptiveness, give it a catchy name, and then advertise sites as conforming to it.

</comment><comment><USER69> It's about time google crack down on these ads. The download button is just one of the many tactics. Any ads that show a guy with biceps three times the size of his head should also be banned. Yahoo is infested with these "sponsored" ads, and pretty much any other site that lives on ads revenue only.

<comment><USER20>
        Pretend to act, or look and feel, like a trusted entity — like your own device or browser, or the website itself.

These are deceptive tactics, intended to confuse and trick.

What you describe is annoying and perhaps exploitative of humans desires, but in the end no different from normal advertising and far less evil than the ads described in the quote above. You want a full ad-blocker. Perhaps you're already using one and want to justify it.

</comment></comment><comment><USER70> There's an unexpectedly large number of commenters who seem to think that this falls under free speech. Do I need to explain that the crap these ads usually download when clicked is responsible for a ton of support calls, many of which go to innocent kids on weekends just trying to unwind from school? :)

I'm kidding of course but seriously comparing blocking these buttons and deceptive elements is not censorship, it's Google saying to these publishers that if they don't get their shit together, that they will dissuade traffic from visiting their sites. The only way to get the attention of bigger publishing companies is to grab them by the revenue stream, you all know this.

</comment><comment><USER71> As I write this, the ranking of the comments here is... strange. Those who see this as being yet another way of Google using their power to manipulate what people see on the Internet are being heavily downvoted, while those agreeing with the practice are not? That doesn't feel like HN to me.

I'm in the former group. This mollycoddling is just going to lead to more users who can't decide for themselves whether something is suspicious or not and are thus easier to deceive, which might be exactly what Google wants, but I certainly do not think it is good for the Web as a whole (or even society in general.) Being able to make these sorts of decisions of trust is an important part of growing up in general, and I'd even say "finding the right download button" could be considered a sort of right of passage to being an effective _user_ of the Web, and not just a consumer.

<comment><USER72> You can't seriously be saying that people potentially running into malware because they couldn't figure out with download button was the real one is reasonable?

<comment><USER71> You can't seriously be saying that not being able to figure out which download button is the right one is reasonable? With experience, it's _extremely_ easy to find the real one.

\- It's usually smaller and less prominent than the fake ones.

\- Mousing over it doesn't show a huge long URL to some external domain that sounds ad-like.

Using adblock probably gets rid of a lot of the fake ones too, but the general principle here is if it looks too good/easy to be true, it probably is. The buttons that seem really enticing are the ones you _don 't_ want to click, and it's that odd, not-very-attractive one that you want.</comment></comment></comment>


Ask HN: What is the reason that many startups end their company name in "ly"? - <AUTHOR>

Just trying to understand if there's some good thinking behind this or if it's just a fad whose time will come and go and date a company. <comment><USER2> I think it's a little of both - driven by a lack of available good .com/net/org domains and by the increasing popularity of sites like bitly. It's little like how startups are jumping on -ify naming fad.

</comment><comment><USER3> It makes me think of Ned Flanders' "oke-ly doke-ly." I don't see it as particularly clever, but then again de gustibus non est disputandum.

</comment><comment><USER4> \- 'ly' is used in English as part of many adjectives.

\- There's a CCTLD for Libya called 'ly' with domains available.

<comment><AUTHOR> But most of the company names are not using 'ly' in the adjective / adverb form? I don't get it. Why not use any other modifier ending, like 'ing' or 'ish' or 'vore' like:

Chargish Bitvore Spoting

If it's the availability of domains and the CCTLD, why Libya? Wouldn't Liechtenstein be better? It ends in 'li'. Or even better, Switzerland, since you can make so many more words that end in 'ch'?

eatmu.ch cin.ch

Puzzled.

Seems so random to me.

<comment><USER4> \- Why not use any other modifier ending, like 'ing' or 'ish' or 'vore' like:

See point #2 - I bet if .ish was a CCTLD they'd do exactly this.

\- Libya? Wouldn't Liechtenstein be better? It ends in 'li'.

See point #1. :^) Pardon if it wasn't obvious but both these facts combined for handy 'sniglet' (fake but realistic English word) domain names.

<comment><AUTHOR> How about .ch -- doesn't it address points #1 and #2? Anyways, you did a good job answering my question. I guess it's really just the intersection space in the Venn diagram where set #1 is A Good/Memorable Company Name and set #2 is An Available Domain Name. I guess Libya was fortuitous in that way. I think Switzerland might make a better choice, but that's me.

</comment></comment></comment></comment><comment><USER5> It's a fad, nothing more than that.

</comment><comment><USER6> Just a trend.

</comment><comment><USER7> jeah its just a name affix

</comment><comment><USER8> Well from where I see it, I believe it's because a lot of .com, .net, .org domains are taken.

For example, if your startups name was Twitterly, you would choose Twitterly.com, but obviously it would be taken, so they chose to use Twitter.ly

That's what I believe.

</comment><comment><USER9> why not just tack "-olovich" on the end of everything, that's what we did when we were 10yrs old.</comment>


If this is NSFW, you should quit - <AUTHOR> http://swizec.com/blog/if-this-is-nsfw-you-should-quit/swizec/4877

<comment><USER2> definitely nsfw

<comment><USER3> I concur. Definitively NSFW and I shouldn't have trusted it at work.

<comment><USER4> ditto</comment></comment></comment>


Open Sourcing Mental Illness - <AUTHOR> https://osmihelp.org/ <comment><USER2> They link to Zero Suicide, which is good, but they should also link to Suicide Safer Communities which is possibly more effective to reduce rates of suicide because it recognises that suicide prevention is everyone's business, and that suicide isn't limited to people with diagnosed mental illness.

[https://www.livingworks.net/community/suicide-safer- communit...](https://www.livingworks.net/community/suicide-safer-communities/)

<comment><USER3> At this point I have to say that if anyone in NSW, Australia is thinking of suicidal thoughts - do NOT call the police, or anyone in the Public Health Service. This absolutely includes the Ambulance Service.

Why? Because of the following:

      20 Detention on information of ambulance officer [1]

      (1) An ambulance officer who provides ambulance
      services in relation to a person may take the person
      to a declared mental health facility if the officer
      believes on reasonable grounds that the person appears
      to be mentally ill or mentally disturbed and that it
      would be beneficial to the person's welfare to be dealt
      with in accordance with this Act.

      (2) An ambulance officer may request police
      assistance if of the opinion that there are serious
      concerns relating to the safety of the person or other
      persons if the person is taken to a mental health
      facility without the assistance of a police officer

Then section 21 defines "police assistance":

Police assistance [2]

      (1) A police officer to whose notice a police assistance
      endorsement on a mental health certificate, or a request
      for assistance by an ambulance officer under this Division,
      is brought must, if practicable:

      (a) apprehend and take or assist in taking the person the
      subject of the certificate or request to a declared
      mental health facility, or

      (b) cause or make arrangements for some other police
      officer to do so.

      (2) A police officer may enter premises to apprehend
      a person under this section, and may apprehend any such
      person, without a warrant and may exercise any powers
      conferred by section 81 on a person who is authorised
      under that section to take a person to a mental health
      facility or another health facility.

I'm sure people in the U.S. Can see how problematic this is. And - it's not just an abstract concept to me, it happened to me several months ago.

1\. [http://www.austlii.edu.au/au/legis/nsw/consol_act/mha2007128...](http://www.austlii.edu.au/au/legis/nsw/consol_act/mha2007128/s20.html)

2\. [http://www.austlii.edu.au/au/legis/nsw/consol_act/mha2007128...](http://www.austlii.edu.au/au/legis/nsw/consol_act/mha2007128/s21.html)

<comment><USER4> When the police get involved with mental illness in the US, you're at risk of ending up dead: [http://www.miamiherald.com/news/local/crime/article90905442....](http://www.miamiherald.com/news/local/crime/article90905442.html).

<comment><USER5> I once overheard: "Not sure if you want to take your own life? Call the police and they'll help that right along."

</comment></comment></comment></comment><comment><USER6> I liked their previous name "devpressed" better. But good initiative.

My personal piece of advice for any developer suffering from anxiety, depression or other mental problems, is to focus on posture. Bad posture can have a really big impact on mental health. You can have RSI in your upper back without knowing it (arms & hands still working okay), and this can cause all kinds of other problems.

<comment><USER7> I mean, really? Reminds me of something Dr. Steve once said:

[https://www.youtube.com/watch?v=K8zOlpPUDCg](https://www.youtube.com/watch?v=K8zOlpPUDCg)

</comment></comment><comment><USER8> There's also a UK startup doing similar things, commercially:

[https://sanctus.io/](https://sanctus.io/)

Both seem well-intentioned but make me deeply uncomfortable. I suppose the fundamental reason being that if you want to destigmatize mental health issues, maybe you should destigmatize seeking treatment from mental health professionals first.

<comment><USER3> If the mental health professional then decides you should be involuntarily committed or detained in a mental health facility, you're screwed.

I applaud the motives behind #itsoktotalk but if you _do_ talk and they strip you of any decision making, then this campaign will fail. The result of talking about thoughts of suicide very often leaves the sufferer in a mental health facility - or waiting for hours in a stressful ER room (or worse) and you won't be seen to at all.

The hashtag should be #itsnotoktotalk

</comment></comment><comment><USER9> I don't understand what makes this open source. It seems to be a nonprofit for the destigmatization of mental illness by a forum to discuss online and I guess resource collection. By that logic, several craft hobby communities are open source.

</comment><comment><USER10> I'm having trouble finding the source code, am I the only one?

I wonder what a mental illness compiler would look like. Would every user have to interact with a ML system to derive their own opcodes? Could it be used both for curing and causing mental illnesses?

"The next 3 months I'm taking a sabatical. I will go to a resort where I will be given clinical depression.

A childhood friend died of it when there was no cure, and I've always wondered what her experience was like.

I hope understanding what she went through will give me some closure."

<comment><USER11> > I'm having trouble finding the source code, am I the only one?

[https://github.com/osmihelp](https://github.com/osmihelp)

</comment></comment><comment><USER12> Well, that's an unfortunate name.

My first thought was that now I can get a mental illness for free. Um, no, thanks.

<comment><USER13> My first thought was, this is great.</comment></comment>


Ask HN: Moving from tech to healthcare? - <AUTHOR> Hi HN,
Until recently, I've been employed as a software engineer for 2.5 years and am now considering shifting careers into healthcare. Although coding is fun, I'm struggling to enjoy sitting at a desk all day, and I really enjoy interacting with people.
I'm hoping to utilize my technology background (Computer Engineering and Computer Science double major) at least a bit, while still interacting with people, possibly understanding complex diseases or health system issues.
I'm looking for the following:
- ideas for career paths
- advice from people who have gone through similar transitions
- advice from people in healthcare that use an engineering background but aren't necessarily engineers in their day to day
I'm considering looking into translational medicine master programs (e.g. UCSF MTM), but I think at the end of the day, I still want a more clinical oriented path.
My resume if you want to learn more about me:
https://joer14.github.io/JoeRowleyResume.pdf
Thanks in advance for your thoughts! <comment><USER2> I would run run run away from healthcare (in the USA, at least, unless you are going to be a doctor or open your own practice where you get to be the boss).

My wife has worked in the field for years and the way they treat people is horrific. No respect for the individual worker, big focus on making numbers (including sometimes some shady stuff), a cog in a big machine.

It was very dehumanizing, and every time I told her about how someone at my work indicated their respect for me, she would respond with some story of disrespect. And don't get me started on the IT/software.

Now, she worked at larger organizations in the USA and was a highly trained medical professional, but not a doctor. I'm sure it's different some places, but if you must, I'd shadow as much as you could before making any big moves.

<comment><AUTHOR> Hey,

Thanks for taking the time to respond.

I'm sorry to hear about your wife's experience - no one deserves that. There are few industries with as much intensity as healthcare - when you get the ego of some doctors, pressure from raising costs, combined with very serious healthcare issues, you get a place ripe for abuse and shitty team dynamics. I like to be optimistic and hopeful that this doesn't have to be the status quo.

Does your wife regret her career in healthcare? What do you think she would change if she were able to go back in time?</comment></comment>


Eye-Fi cards will be able to upload photos to your smartphones and tablets - <AUTHOR> http://venturebeat.com/2011/01/04/eye-fi-cards-will-be-able-to-upload-photos-to-your-smartphones-and-tablets/ <comment><USER2> Pretty sure the Pro X2 cards already had Ad-hoc mode enabled. Does this mean they're adding this feature to the standard model?</comment>


New Kindle Wifi: $139 3g: $189 - <AUTHOR> http://www.amazon.com/ <comment><USER2> This is a nice update and I will finally buy one, although the DX would be better suited for my needs, the price isn't quite right yet.

It should be mentioned that the new Kindle will arrive in the UK on August 27th as well. Should be good news for all EU customers. (Apparently can't ship outside the UK...)

£109.00 (~134EUR) Wifi-Only and £149.00 (~180EUR) 3G</comment>


Why Percentiles Don't Work the Way We Think - <AUTHOR> https://www.vividcortex.com/blog/why-percentiles-dont-work-the-way-you-think <comment><USER2> Quantile Digest (Q-digest) or something similar is what I believe is desired here.

From what I understand it's a fixed size data structure, represents quantiles as a tree of histogram bands, pruning nodes with densities that vary the least from their parents to achieve error / size trade-off. They also have the property that you can merge them together and re-compress in order to turn second data into minute data, or compress more accurate (large) archival digests into smaller ones to say, support stable streaming of a varying number of metrics across a stream of varying bandwidth by sacrificing quality.

They're pretty simple because they're designed for sensor networks, but I think you could design similar structures with a dynamic instead of fixed value range, and variable size (prune nodes based on error threshold instead of or in addition to desired size).

If anyone knows of a time-series system using something like this, I'd love to learn about it.

</comment><comment><USER3> You know what I've always found really useful? Entire distributions. Forget means, medians, percentiles, etc. — give me the complete distribution (along with sample size) so I can understand all of the nuances of the data.

(Better yet, just give me the raw data so I can analyze it myself. I find it hard to blindly trust someone else's conclusions considering all of the p-hacking going on nowadays.)

<comment><USER4> Yeah. We're starting to contemplate a new monitoring/performance/etc. system, and I'm thinking I don't want to do any of the long-term storage aggregration like in rrd. We'll store the full data for each item forever. Not sure if we can afford the storage, but I want to try for a while and see if it works.

<comment><USER5> We did something similar at Netflix. We had all the aggregations but also stored all the raw data. The raw data would be pushed out of the live system and replaced with aggregates and then stored as flat text in S3. If for some reasons you needed the old data, you just put in a ticket with the monitoring team to load the data back into the live system (I think this is even self service now).

The system would then load the data from S3 back into the live system via Hadoop. Turns out it was pretty cheap to store highly compressible files in S3.

<comment><USER6> I've built a similar system as well. Raw data compressed and stored in s3, but aggregations are stored in postgres. Data stored in Postgres is compressed binary representation of a histogram and I added few C functions in postgres to do things like select x,y, histogram_percentile(histogram_merge(data), 99) group by x,y etc..

</comment></comment></comment></comment><comment><USER7> I've recently written a page about "averaging" percentiles correctly by approximating the combined histograms of two distributions. This is demonstrated in a live time diagram with logarithmic time axis here:

[http://www.siegfried- kettlitz.de/blog/posts/2015/11/28/linlo...](http://www.siegfried- kettlitz.de/blog/posts/2015/11/28/linlog_plot_quantiles/)

If you have questions or comments, feel free to reply or email.

</comment><comment><USER8> A more accurate title would be something like: "Percentiles are great, but you may not be calculating them correctly."

</comment><comment><USER9> Great article, I ran into this exact problem at work today. You can also get a very similar problem if the timeseries aggregation system you're using does any pre-aggregation before you calculate the percentile - for example, if you sample your servers every 90 seconds, then any latency number it reports is likely already averaged over the requests the server received during that time period, meaning your 99th percentile number is really the latency of the 99th percentile server, not the 99th percentile request. Using latency buckets solves this problem as well, however.

</comment><comment><USER10> The article is useful because it outlines many different ways in which to monitor the performance of a system, many of which are better than just looking at the mean and P99. However, the main thesis that "an average of a percentile is meaningless" is just plain wrong. If the distribution is fixed, then averaging different P99 measurements will give you the best possible estimate of what P99 is in the population (as opposed to your sample.) If the distribution is moving (you're making performance improvements or your user base is growing) then a moving average of a percentile will move with it.

<comment><USER7> You are right when looking at the true P99 of the whole distribution. Then the P99 will always have the same value. The problem arises when estimating a distribution from incomplete data.

E.g. estimating the P99 from 200 data points each will give you a large variation. In this case averaging will be wrong. This gets worse if you experience bunching. E.g. a server will have a high ping on several packets when the load is high; after that it will have a low ping for thousands of packets when the load is low. Here, it just doesn't work.

Instead, you can get a better result by adding the (estimated) histograms and resampling the quantiles from the combined distribution function.

Here I've written how that works with a live demonstration: [http://www.siegfried- kettlitz.de/blog/posts/2015/11/28/linlo...](http://www.siegfried- kettlitz.de/blog/posts/2015/11/28/linlog_plot_quantiles/)

<comment><USER10> Even P99 estimates averaged from many tiny samples are at most inaccurate, they won't be biased.

The problem you talk about is one of granularity: you don't want to know how bad things are in general, you want to know how bad things are when they are actually bad. That's perfectly legitimate, but it's a different metric altogether that just happens to also be based off of the 99th percentile. It depends on your definition of "the distribution".

This probably sounds like nitpicking, but people should know what they're talking about when they call out others for being wrong.

<comment><USER7> > Even P99 estimates averaged from many tiny samples are at most inaccurate, > they won't be biased.

Are you sure about this?

<comment><USER10> Sure, simulate it if you don't believe me.

        x <- rnorm(1000, mean=50, sd=10)
        quantile(x, 0.99)
        mean(sapply(1:20, function(i) {
          i <- i * 50
          quantile(x[i-49:i], 0.99)
        }))

<comment><USER7> Matlab / GNU/Octave code:

        rand('seed',123);

        outer_rounds = 100;
        inner_rounds = 100;
        items = 500;

        q99_mean_vec = [];
        q99_complete_vec = [];

        for ior = 1:outer_rounds;

          q99_vec = [];

          x = rand( 1, rounds*items );

          for iir = 1:inner_rounds

    	q99_vec(iir) = quantile( x( (iir-1)*items+(1:items) )', 0.99 );

          end

          q99_complete_vec(ior) = quantile( x', 0.99 );
          q99_mean_vec(ior) = mean( q99_vec' );

        end

        hist( [ q99_complete_vec; q99_mean_vec ]', [0.988:0.0001:0.991] );
        xlim( [0.988, 0.991] );
        legend( 'from complete data', 'from means' );

This looks biased to me even though the distribution is uniform. I did not expect that. Maybe the quantile function is broken.

EDIT: I've installed R and run your code. You use a very low number of samples in the mean, so I changed it a bit. 1. I use the uniform distribution, so we know the true value of the 99th percentile is 0.99. 2. I've increased the number of samples:

      x <- runif(100000)
        quantile(x, 0.99)
        mean(sapply(1:2000, function(i) {
          i <- i * 50
          quantile(x[i-49:i], 0.99)
        }))

When run multiple times, I've found that the mean is always lower than the quantile over the full sample.

<comment><USER10> Actually, doh, you're right, my bad, turns out sample quantiles are only asymptotically unbiased, and a better estimator would take some sort of weighted average of P99 and P99.5 with the weights depending on the sample size. (You still don't need the full population and you don't have to merge histograms, though.)

<comment><USER7> I find this part of statistics very hard to argue about. Simulations only work with well-known distributions, but even then not all of them. Even when you have a simulation that confirms hypothesis A1 using distribution D1 it says little about some pathetical distribution D2.

Using mathematical precision might take years to get it right and often there is no easy answer.

What I've been looking for is something that is fast and works "good enough". Estimating the 95th percentile and averaging it did not work well in my use case. Using the histogram method does work well although it certainly is not perfect.

<comment><USER11> Nonparametric estimates of the CDF have a long history and plenty of theory: [http://www.win.tue.nl/~rmcastro/AppStat2013/files/lecture1.p...](http://www.win.tue.nl/~rmcastro/AppStat2013/files/lecture1.pdf) [http://www.mathworks.com/help/stats/examples/nonparametric-e...](http://www.mathworks.com/help/stats/examples/nonparametric- estimates-of-cumulative-distribution-functions-and-their-inverses.html)

<comment><USER7> Thanks, this is informative and pretty similar to what I've been implementing in JavaScript. I guess I'll have to take a deeper look into literature when doing an update to that topic.

<comment><USER11> You may also be interested in [http://www.amazon.com/All-Nonparametric- Statistics-Springer-...](http://www.amazon.com/All-Nonparametric-Statistics- Springer-Texts/dp/0387251456) and [https://en.wikipedia.org/wiki/Bootstrapping_(statistics)](https://en.wikipedia.org/wiki/Bootstrapping_\(statistics\))

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER12> Gave the post an upvote because it is interesting from a theoretical perspective, but I have a hard time imagining a real life scenario where averaging a 99 percentile will lead you to the wrong conclusion.

Perhaps I'm wrong, but whenever I'm looking at the tail of a distribution, it's usually just to understand the order of magnitude, not to reach a precise number.

</comment><comment><USER13> The problem with percentiles is that you are discarding data points. e.g. Measuring the 98th percentile involves discarding the top 2% of data.

The problem with that is, sometimes the top 2% you are discarding might correlate to your top 2% of customers...and you are literally throwing their data away by using percentiles. Not good.

My recommendation is to pick two aggregation types. Maybe Percentile and Maximum, or Maximum and Mean, or Percentile and Mean. You can't really go wrong with that approach.

</comment><comment><USER14> Does anyone know the math behind exactly how wrong the averaged percentiles are? My dim understanding of stats makes me think the central limit theorem is at play here; the averaged p99 values will tend towards a normal distribution, which is obviously wrong. Would love to be schooled on it.

</comment><comment><USER15> I think the lesson is, don't just blindly calculate some numbers / metrics, have a look at your data (visually!) and see if it makes sense (for instance do you use the 99th / 95th / 90th percentile).

</comment><comment><USER16> A day late, but I wanted to add: at New Relic (where I work) we ended up just deciding to store all the data points. We literally store every transaction and page view for our customers and then go back and read every data point when they are queried.

</comment><comment><USER17> Why not let the user decide? Let them switch the graph between average, median, 95%, 99% and histogram. And everyone will be happy!

</comment><comment><USER18> Headline is utterly false.

<comment><USER19> I mean, "Why Percentile Calculations Don't Work the Way you Think" is better, but I don't think the current one is "utterly false".

<comment><USER20> It depends who's reading it. The whole genre of titles "* you don't know", "you're wrong about *" is condescending and frequently false.

I suspect that's what the parent post was referring to and I would agree: percentile calculations do work the way I think, and the article didn't say anything controversial.

<comment><USER21> I'm the article's author. I don't think the headline HN chose is great but I don't mind if people have different interpretations of what I write. Hopefully all the commenters here are actually reading the original article and judging it on its merits (and flaws), right? right?</comment></comment></comment></comment>


Authenticate on OS X with iPhone Bluetooth LE - <AUTHOR> http://guoc.github.io/nearbt/ <comment><USER2> Doesn't MacID already do this with Proximity Wake?

[Http://MacID.co]

<comment><AUTHOR> This doesn't require system passwords to be stored somewhere.</comment></comment>


Uber's real time car map is fake - <AUTHOR> http://www.wired.co.uk/news/archive/2015-07/28/uber-algorithm-fake

<comment><USER2> I'm not really sure extrapolating that from surge pricing is a valid way to study it.

<comment><USER3> Thats not whats going on, they just mentioned surge pricing itself was based on extrapolation.

<comment><USER2> Looking back at it, I may have misread. <3</comment></comment></comment>


Ask HN: What's the best way to promote your app? - <AUTHOR> <comment><USER2> Ah, if only there was a 'Best way' \- but there isn't. Indeed, the nature of marketing is that most successful promotional techniques are quickly replicated by others and their effect diluted.

Instead, ask yourself: 1) Who are our clients / users? 2) What are their pains, how does our app solve those, and why would they buys/use our app? 3) Where are those clients - and therefore how do we get in front of them?

As an extreme example, the best way to promote a fashion app for 16-year-old girls will be quite different to a finance app for high net worth individuals.

I heartily recommend the book Traction for a thorough list of options and how to approach them. (The list is included in the free preview, if you're strapped for cash, but the extra information is worth buying a copy.)

Best of luck!

<comment><USER3> I'm finding several books on Amazon, can you provide a link to the book you're recommending?

</comment></comment><comment><USER4> In my opinion, the best way to promote your app is with word of mouth advertising. Unlike other forms of advertising, you can't buy word of mouth advertising; instead you need to build something that solves real problems and creates true value for your customer. The bigger problem you solve, the more value you have. A high perception of value builds loyalty, and lays the groundwork for customer advocates (which are your most important customers). When your customer talks with other potential customers that have the same problems, your app or service becomes a potential solution. Word of mouth advertising is powerful because recommendations from friends and peers carry more value than traditional advertising. Potential customers are more likely to try your app or service when it's been recommended by friends or peers. This is most effective when it occurs naturally (not forced). These are lessons learned from my first startup (results of our product, customer reactions, customer surveys, etc)

</comment><comment><USER5> Check out this app launch guide on GitHub: [https://github.com/adamwulf/app- launch-guide](https://github.com/adamwulf/app-launch-guide)

</comment><comment><USER6> It depends, if your target audience is early adopters submit the landing page of your apps to SHOW HN and Product hunt is a good start

</comment><comment><USER7> The one by Gabriel Weinberg and Justin Mares?

<comment><USER2> Yes - I should have linked to it [http://tractionbook.com](http://tractionbook.com)

<comment><USER7> Thanks

</comment></comment></comment><comment><USER8> Well, We wrote this, Have a read :)

[https://news.ycombinator.com/item?id=9550294](https://news.ycombinator.com/item?id=9550294)

<comment><USER9> The link opens into a blank HN page.

</comment></comment><comment><USER10> hire a pr agency</comment>


Here's an online gift-registry I created. What do you think? - <AUTHOR> http://www.giftlisting.net

<comment><USER2> Poor font. Makes the text hard to read. Too little information. Why should I sign up? How is this different from Facebook gifts/Amazon Wish list? An intro video/slideshow will be nice!</comment>


How to get started with advanced Lego building - <AUTHOR> http://swooshable.com/how-to-start-building-lego#explore <comment><USER2> Wow this is awesome and what I have been looking for.

I've loaded up on the $40 1,500 brick sets from Costco :) Now time to waste the weekend.

</comment><comment><USER3> brings back memories of ldraw. discovered the multifaceted precision of lego pieces from ldraw wiki and by reading the ldr files.

had only a handful of actual store-bought lego sets, but almost the entire catalog was on my pc, it was amazing.

ldraw+mlcad+ldview: hours of fun!</comment>


Wood gas vehicles: firewood in the fuel tank - <AUTHOR> http://www.lowtechmagazine.com/2010/01/wood-gas-cars.html <comment><USER2> Urea/diesel hybrids would be fun, since every modern engine has to use urea for SCR anyway.

</comment><comment><USER3> Mother Earth News had an article about this: [https://www.motherearthnews.com/green- transportation/green-v...](https://www.motherearthnews.com/green- transportation/green-vehicles/wood-gas-zm0z12amzroc)

</comment><comment><USER4> Politicians in Sweden after WW2 had interesting mindset. They really tried to prepare the country for various worst-case scenarios. Researching wooden gas for cars was one thing.

Another is how they investigated if they should make own nuclear weapons. They decided not to after getting assurances from industry that they can make an atomic bomb within months if needed.

I wonder what other possibilities or backup technologies they considered?

</comment><comment><USER5> I find myself pondering the efficiency of removing the gasifier from the vehicle and instead handling it similar to LNG is today.

</comment><comment><USER6> This will make for an excellent premise for a post-apocalyptic story, where eventually the organic biomass starts being over consumed. Basically "Soylent Green" but with cars.

</comment><comment><USER7> There was a talk from the Royal Institution just uploaded yesterday (talk was in June), _The Apocalypse and How to Avoid It_ [0], where Lewis Dartnell (author _The Knowledge_ [1]) describes a gasifier stove you can make from a couple bean cans, and also brought up this gasifier engine technology and how it was used in London during the Blitz. This article has a ton of useful information that makes for a great followup.

Also at the talk were Vinay Gupta (Ethereum), Rosalind Eggo, and Hugh Lewis discussing different avenues to potential Apocalypse and their likelihood, etc.

[0] [https://www.youtube.com/watch?v=FPxBhqonZEQ](https://www.youtube.com/watch?v=FPxBhqonZEQ)

[1] [https://www.amazon.com/Knowledge-Rebuild-Civilization- Afterm...](https://www.amazon.com/Knowledge-Rebuild-Civilization-Aftermath- Cataclysm-ebook/dp/B00DMCV5YS/)

<comment><USER8> As famously seen in dads army when they converted Jones the butchers lorry.

</comment></comment><comment><USER9> I'm not saying this is common knowledge, but it's not a new idea: [https://archive.org/details/FEMAEmergencyGassifer](https://archive.org/details/FEMAEmergencyGassifer) , [https://www.nrel.gov/docs/legosti/old/3022.pdf](https://www.nrel.gov/docs/legosti/old/3022.pdf) . Then again, the fact that the article focuses on WWII cars says it's not a new idea.

<comment><USER10> Could we make it better though? a lot of heat must be lost gassifying the wood? Perhaps we could use that for thermometric power generation too.

<comment><USER11> Possibly. But harvesting that heat directly to 'motion' would probably not be feasible. Storing that on batteries may be better, as they can be slowly trickle charged, but provide a lot of power when there is demand.

</comment></comment></comment><comment><USER12> I saw some of these in use while visiting DPR Korea. We weren't up close, but the government tour guide confirmed that they were wood-burning trucks. Sadly I was not allowed to photograph the trucks. Like the article mentions, deforestation is an issue they can cause, something Korea suffers from greatly.

<comment><USER13> When resources were scarce during WW II in Germany there were various kinds of wood gas vehicles. Here is a video of a 1942 tractor:

[https://m.youtube.com/watch?v=RMVwbjp514o](https://m.youtube.com/watch?v=RMVwbjp514o)

<comment><USER14> From what I heard, it was the same in France, with various vehicles converted to use wood gas too.

</comment></comment></comment><comment><USER15> Important downside explained half way through: "if we were to convert every vehicle, or even just a significant number, to wood gas, all the trees in the world would be gone and we would die of hunger because all agricultural land would be sacrificed for energy crops."

<comment><USER16> In the last years of WW2 I think Japan chopped down the last of its trees to make a bit of aviation fuel.

<comment><USER17> Wait, what? There's absolutely no evidence that "Japan chopped down the last of its trees". Why would you say such a thing? You enjoy lying on the internet?

<comment><USER16> I saw it in the documentary series "The world at war". I think the episode about Kamikaze fighters. Japan had virtually no oil supplies. Here's an article about it. [http://thenexthurrah.typepad.com/the_next_hurrah/2006/05/the...](http://thenexthurrah.typepad.com/the_next_hurrah/2006/05/the_japanese_ma.html)

Ok they don't chop down all of their trees. Of course they still have trees.

</comment></comment></comment></comment><comment><USER18> Would a steam engine make more sense? Then you could run it on anything.

<comment><USER19> In order to be efficient and powerful, a steam engine needs high-pressure steam, which can only be safely contained by thick, heavy boilers and pipes. This makes miniaturization of steam engines for cars difficult. Gasification of wood allows you to keep the benefits of the internal combustion engine.

<comment><USER20> I think the bigger issue is adapt vs rebuild.

The Doble steam cars are at least on the cusp of practical.

(they are very high maintenance compared to modern cars)

<comment><USER21> Thanks; I had never heard of the Doble Steam cars until now. I found this video of Jay Leno driving one.

[https://www.youtube.com/watch?v=rUg_ukBwsyo](https://www.youtube.com/watch?v=rUg_ukBwsyo)

</comment></comment></comment></comment><comment><USER22>
      coal monoxide is a slow burning gas

I've never seen anyone refer to CO as "coal monoxide" before.

<comment><USER23> It looks like the author's first language may be Dutch, where carbon monoxide is "koolmonoxid" and "kool" can mean both "coal" and "carbon" (although there are more specific words for coal such as "steenkool" = "stone carbon").

<comment><USER24> Same in most Germanic/Scandinavian languages I believe. Carbon and Coal are synonymous.

Coal as in coal power, coal barbecue, and the element carbon use the same word. That's not a coincidence.

Apart from some exotic forms like diamonds, trees or rabbits (less pure), coal is how we normally encounter carbon.

<comment><USER25> In German "Kohle" means coal, and "Kohlenstoff" is carbon. It's not the same word.

(But carbon monoxide is "Kohlenmonoxid")

<comment><USER26> But "Kohlenstoffmonoxid" is the actual correct term. [https://de.wikipedia.org/wiki/Kohlenstoffmonoxid](https://de.wikipedia.org/wiki/Kohlenstoffmonoxid)

</comment></comment></comment></comment></comment><comment><USER27> Nice thing about wood power is that as long as you have a sustainable source of wood, you're 100% solar-powered _and_ carbon neutral. That is to say, it's a source of energy that is (can be) non-fossil, nor derived from any fossil sources, and which recycles it's own CO2 emissions.

There are very few other alternatives that can claim the same. Perhaps Hydro? Solar panels don't offset their construction CO2 costs, and in fact almost can't even pay back their own construction energy costs if you factor in batteries plus materials, mining, and shipping.

<comment><USER28> >Nice thing about wood power is that as long as you have a sustainable source of wood, you're 100% solar-powered and carbon neutral. That is to say, it's a source of energy that is (can be) non-fossil, nor derived from any fossil sources, and which recycles it's own CO2 emissions.

As long as you're happy pouring carcinogens into the air, sure. Also, compared to oil or coal there's not much energy in wood, so it doesn't scale very well.

<comment><USER29> There's a lot of energy in wood compared to coal. Wood has around 2/3 of the energy of coal by mass. What do you think coal is made from?

<comment><USER30> Isn't most of the mass of wood water?

<comment><USER31> Dried wood suitable for burning is around 20% water at most, else it won't burn with much heat. It can go down to around 6% IIRC for kiln-dried wood suitable for construction.

<comment><USER32> Boiling water costs a helluva lot of energy. Do these engines recover that energy?

<comment><USER33> How would they work if they didn't?

<comment><USER32> At low efficiency.</comment></comment></comment></comment></comment></comment></comment></comment>


How much data do you think Google collects from it's free services? - <AUTHOR> How much data do you think Google collects from it's free services like DNS, CDN (Google Fonts) etc. I'm not referring to obviously data driven services (email, search, analytics, maps, translate, captcha etc), nor services that have a  clear commercial purpose (Chrome, Android) <comment><AUTHOR> I see nothing wrong in collecting data from the services that have a clear commercial intent: search, email, maps,captcha,chrome,android etc.

But I would have problems if Google where to log (and keep) queries to 22.214.171.124 or utilize that data in any way.

</comment><comment><USER2> Everything

No more no less

<comment><AUTHOR> But their DNS service is recommended to everyone in oppressed countries, routers have it by default, Linux distro's (even security oriented ones) have it by default... ..and Open Fonts and other projects hosted on their CDN... They are the champions of a better web, making the web better, faster, safer, more beautiful.

Do you really think they collect and store all that data? All the logins, all the DNS queries, all the hits to the CDN?

<comment><USER2> It is how google works, make stuff freely available, collect data, use it to make money by showing targeted advertisements</comment></comment></comment>


Show HN: Prophecy.io – Cloud Native Data Engineering - <AUTHOR> https://medium.com/prophecy-io/introducing-prophecy-io-cloud-native-data-engineering-1b9247596030 <comment><AUTHOR> Hi Everyone! Super excited for our first release!!

I was product manager for Apache Hive at Hortonworks and earlier an early engineer at CUDA at NVIDIA - focusing on compiler optimizations.

As product manager, I saw so many customers struggle with Hive/Hadoop. So, I decided to build a product first company - that just works to solve real (and sometimes unsexy) pain points of customers.

We want to support entire Enterprise journey to Open Source (Apache Spark) and then to cloud (Kubernetes).

What I'm personally excited about is that with some compiler magic - I can make code and visual interfaces both work together - making all developers happy!

I'd love to hear what you think, what you wish we'd build, and I'll be here to answer any questions!

<comment><USER5> This looks very compelling. I'd love more to read about it. Having suffered in various visual ETL trenches I've pretty much walked away as much as possible. When can we see the source?

<comment><AUTHOR> I'd love to understand what your preferred interface looks like, and get your input in building the interface, please send me a mail at email@example.com

</comment></comment></comment><comment><USER6> Very cool that you can go between drag-n-drop and code for development. How is Prophecy different from datacoral.com or Astronomer.io? Will it be open sourced (ex, dbt, airflow, Dagster)?

<comment><AUTHOR> Foremost, we're focused on Transforms - helping users create and manage complex ETL transforms - this is your business logic.

One is the category of schedulers - airflow, astronomer - these are focused on scheduling these Transforms, so these are very different (we integrate well with airflow).

Then there is the NoCode category of transform services on cloud - these focus on simple movement of data - datacoral, segment, fivetran and all might fit here.

We're quite different from both!

</comment></comment><comment><USER7> Out of personal (and painful) experience, using Spark for general-purpose ETL processes is a bad idea. Spark is meant to be used in highly-distributed systems and with tables that have like trillions of rows. RDDs use an optimized distributed programming model that takes a lot of practice and getting used to. Some operations are virtually impossible due to the fact that executors run on separate contexts. Caveat emptor.

<comment><USER8> What issues did you encounter? We've been using Spark since 1.0, and the dataframes API and Catalyst query engine have abstracted away RDDs nicely, we rarely have to use them. We recently converted a bunch of legacy ETL jobs that ran against the same dataset in Vertica to Spark in EMR by storing the raw data as Parquet, caching it in memory, and running the 30 or so different SQL queries against it using Spark SQL.

Only issue we encountered was one query used three unions which was rather inefficient, but once that was replaced with a query using grouping properly, problem solved.

<comment><USER7> First, like I mentioned, Spark uses separate contexts for all executors (by design). This means that running nested RDD mappings is a nightmare. Just Google "nested for each loop Spark" or something along those lines[1]. This is an _extremely_ common paradigm, especially when dealing with more complex transforms (in my specific case, we needed to hit an API, and then hit another API based on results of the previous API).

Second, Spark is not re-entrant. This means that to really get usefulness out of Spark's massive parallelization, you need to use clever and non-obvious tricks to do something that "seems" simple[2]. In some extreme cases, the RDD needs to be fully serialized (danger zone if it's a big one).

Finally, and related to the two above, Spark is just simply strange. You really need to have a "parallel" mindset when writing code on a Spark cluster. I constantly had to look stuff up and documentation was spotty at best. Once you throw in the rest of the ensemble (Pandas, Numpy, etc.) you end up with a very domain-specific codebase. In reality, most companies will have datasets that could be ETL'd on my iPhone and a Spark cluster is just overkill.

[1] [https://stackoverflow.com/questions/34339300/nesting- paralle...](https://stackoverflow.com/questions/34339300/nesting- parallelizations-in-spark-whats-the-right-approach)

[2] [https://stackoverflow.com/questions/32619570/can-i-use- spark...](https://stackoverflow.com/questions/32619570/can-i-use-spark- dataframe-inside-regular-spark-map-operation)

<comment><AUTHOR> The variation of complexity of transforms across Enterprises and teams within Enterprises is quite large.

Spark and Hive (that I was product manager for, and used by hundreds of Enterprises) are used quite heavily in data engineering. Many transforms fit well into relational/set based model.

In your particular case, perhaps due to the nature of your transforms, it might not have been the best fit.

Also, if your data is small, it's not always a great fit. However, in many large companies, if they have 500 transforms and 30 are small - there is simplicity in using a single technology.

<comment><USER7> Just to be clear, I'm not discounting the work you're doing with Prophecy. We were using Databricks, and I can definitely see how a platform like Prophecy is a step up. I have a personal rule that I will always prop up any budding startup or entrepreneur (the golden rule and all that).

So huge congrats on your release!

</comment></comment></comment></comment></comment><comment><USER9> This looks very similar to products like Apache Nifi, Apache CDAP and Google Data Fusion. When would I pick prophecy.io over the others?

<comment><AUTHOR> We're focused on complex transforms for high volume & performance workloads - something you'd find in Enterprise core production ETL workflows.

We're a complete replacement for ETL products - you'd use us to replace an Ab Initio, Informatica, IBM Datastage - we'd move your workflows through automated conversion.

You can program using visual drag-and-drop or write pure Spark Scala/Java/Python code, they are equivalent in our system. You get configuration management, lineage, metadata management. We're not tied to any private/public cloud - so you can use us with any Apache Spark - <USER2> public/private cloud has it.

This is quite different from these products! Different focus and level of abstraction.

</comment></comment><comment><USER10> How does this compare to cask hydrator/cdap? What does the migration from R or python ETLs to this resemble?

<comment><AUTHOR> We're not adding a layer on top of all big data platforms, instead we work on Spark directly. This has advantages, disadvantages.

When you use our product, your code looks very similar to writing Spark code in IntelliJ. You get visual workflows along with it.

Our users are not having to learn or use another layer of abstractions that locks them into our API, and we avoid performance

Apart from interface, we automatically "convert" your legacy code into Spark & "migrate" these Spark workflows from one cluster to another. You can move workflows from private cloud to public cloud, or to another datacenter.

<comment><USER10> What about existing spark/scala workflows? Can they be imported?

</comment></comment></comment><comment><USER11> I read all this technical jargon but have no idea what any of it actually does.

<comment><AUTHOR> Well, simplest way to put it is - you get an Data Engineering (ETL) product - that'll move your 10s of thousands of ETL workflows from legacy ETL product to Spark, give you a full stack ETL product on Spark, and move you from on- premise to cloud.

Plus it has improvements along multiple dimensions such as interfaces - <USER3> developers care a lot about!

<comment><USER12> what is "legacy ETL" exactly? you have a transpiler but what code does it transpile? what if all my ETL is written in Haskell and SQL (it's not - <USER4> don't have any ETL processes - I'm just trying to figure out what you're transpiling)

<comment><AUTHOR> This is the workflows written in legacy formats - Ab Initio, Informatica, IBM DataStage - this is a $10B+ industry - so lots of usage out there. They typically have graph formats and Domain Specific Languages to specify transformations. We reverse engineer them. Enterprises have 10s of thousands of these workflows and getting out of legacy lock-in is hard!

</comment></comment></comment></comment><comment><USER13> Awesome!

</comment><comment><USER14> That is awesome. I could use your service. Will mail</comment>


Ask HN: Good books for learning to write a fuzzer from the ground up? - <AUTHOR> I want to write a fuzzer from the ground up, mostly for self-educational reasons. I'm interested in both resources that teach the basic algorithms and methods and more modern approaches. <comment><USER2> fuzzingbook.org</comment>


Show HN: Arc – Hire full-time remote developers and engineering teams - <AUTHOR> https://arc.dev <comment><AUTHOR> Hello HN!

We're excited to announce Arc today! We aim to change the way we hire and work remotely.

My first startup SocialPicks (YC 07!) was actually a remote company back in 2006. I was based in the Bay Area, co-founder Keven was in Vancouver, and our lead dev Olivier was in Toronto. This was pre-Slack, pre-Zoom, pre-Trello. I guess we're the OG's of the remote world. Back then it was actually quite hard to grow our remote team, as this would require, essentially, trusting strangers on the internet to join us on the intense rollercoaster ride of startup life.

Fast forward to today: The bulk of remote hiring services are still focused on finding short-term freelancers, rather than thinking of remote work as a genuine full-time employment option. It's clear that the friction of going remote is still too high for many organizations and individuals. This is why we are launching Arc today: to offer a complete remote hiring solution to find full-time remote developers and build remote engineering teams.

We're introducing Arc early to get feedback from the HN community. As we work to reduce the friction of hiring remotely for both sides of the hiring equation, we invite you - developers and organizations alike - to join us on this journey.

We would love to hear any thoughts or feedback!

</comment><comment><USER2> Remote dev here.

I'm signed up to CodementorX for over a year now.

I liked the vetting process and the people working there were very nice and helpful.

Never got a project via that platform, but I also only applied actively for projects for a month or so, then I got a book deal and after I wrote that book I got other projects where companies directly contacted me.

The service itself seems to be rather passive, like, you have to search and apply for projects yourself and hope you get one. It's not that they have a bunch of recruiters doing that work for you, which I would prefer.

<comment><USER3> I've been proactively contacted for CodementorX projects. I believe they favor those who are active and have a good review history in their regular mentoring side of the business.

</comment></comment><comment><USER3> Can you provide some statistics on where your developers are located?</comment>