DuckDuckGo vs Google - <AUTHOR> https://fourweekmba.com/duckduckgo-vs-google/ <comment><USER2> Recently switched over to DuckDuckGo on all my devices. Tried to do the same thing several years ago and found it didn't really work out, but this time around it is so much better, both when it comes to speed and results.

If anyone has doubts because they tried it years ago, I'd say go for it again.

<comment><USER3> And don't forget to use the bangs that can allow you to check other engines results fast if you need.

For people afraid of getting off google, you can always search something like '!g my-search', it works the same for youtube(!yt), google image(!gi), or even hackernews(!hn)

And of course the best bang is the 'I am feeling lucky' one (!), i.e.: 'hackernews !'

<comment><USER4> I'm tired of people mentioning !g. If you want to use google, just use google. Stop using duck duck go just to search google, and then say "but duck duck go is better" lo. no it isn't. clearly.

<comment><USER5> It's a way to wean yourself off of Google or quickly see results from a different perspective. There's nothing wrong with it, it's just an easy way out for people who don't trust ddg just yet.

<comment><USER6> That and its a Good way To help ddg (recording used search queries) and getting google results

</comment></comment></comment></comment></comment><comment><USER7> Those who use DDG, do you miss dates in results? Having a date present definitely helps me think about the results:

[https://www.google.com/search?q=dcss+branch+order](https://www.google.com/search?q=dcss+branch+order)

[https://duckduckgo.com/?q=dcss+branch+order](https://duckduckgo.com/?q=dcss+branch+order)

This isn't a case where I _know_ I only want 2017 results, and so I do the syntax to filter it down automatically. I want all results, but I want to be aware of the timeline of whatever I'm going to click.

But to take the thought further: I can understand when a date isn't important. Say some documentation for a specific programming related thing. You'll probably learn to use !clojuredocs or something.

What about outside that? Those searches I can't quite describe without thinking, but my example above sort of works nicely because that game in particular has changed a bunch (and will continue to) over time and you do care about the date of a forum post or whatever.

For all I know, the answer is "that's when you use !g".

<comment><USER8> The only thing I miss is the "Past Year" option. I find it really important for filtering out programming stuff that's too old. "Past Month" is too short. I can't understand why it's not there. I've sent the suggestion several times.

<comment><USER9> It's because they get results from so many different sources. Many of those sources just don't keep time records or don't offer it through the API that DuckDuckGo interacts with.

<comment><USER10> Time records don't have to be scraped from the site or retrieved from APIs (what APIs btw? Isn't it all scraping?). When the search engine crawls a new page, or notices an update to a previously crawled page, as long as you're crawling thoroughly, those dates can be pretty accurate.

<comment><USER11> DuckDuckGo doesn't just run a web crawler, it aggregates from many sources including Bing and area specific APIs. Not all of these sources supply date information so you'd end up with some results having it and others not.

Information is here: [https://duck.co/help/results/sources](https://duck.co/help/results/sources)

<comment><USER10> Any idea why search engines would allow API access for a competing search engine? Usually APIs come with a caveat that you can't use them to rebuild the same service. Also for ddg's level of traffic, it seems like special deals would need to made, not just a general access key.

<comment><USER12> Weakening Google's monopoly.

</comment></comment></comment></comment></comment></comment></comment><comment><USER13> People are conditioned to think Search has to work online.

This was true when Google was created.

No one had the processing or memory available on their desktop to search an entire index of the "useful" web.

Not anymore.

How large is a "useful" index of the web today? And can it fit on your laptop? The answer is yes.

Can the entire thing be queried fast? The answer is yes.

As an example take the entire stackexchange and wikipedia dumps in their entirety(including images). Compressed it comes to 50-60 GB range. Think about that number. That's an rough approximation of all known human knowledge. It's not growing too fast. It has stabilized. To query the content you need an index.

So how large is an index to a 100 GB file? Generally around 1 GB. Let's say you use covering indexes with lot of meta data and up that to 5GB to support sophisticated queries.

With today's average hardware you can search the entire thing in milliseconds.

So why aren't we building better local search?

Because everyone is conditioned to believe, thanks to Google's success, we need to do it online. Which means baking in the problem of handling millions of queries a second into the Search problem. Guess what? This is not a problem that local search has.

Every time a chimp or a duck needs to build a protein in it's cell it doesn't query the DNA index stored in the cloud. Instead every cell has the index. Every cell has the processing power to query that index in the nanosecond time scale.

The cloud based search story is temporary.

If you want to index every reference to Taylor Swifts ass that every teenager in Norway, Ecuador and Cambodia are making, then yes you need a Google size index. But for useful human knowledge we are getting to the point where we don't need Google scale.

If you don't believe me look at what is possible TODAY in Dash/Zeal docset search for offline developer documentation search or Kiwix or with Mathematica.

<comment><USER14> Your idea doesn't go far enough. What good are a couple of stale indexes of Wikipedia and Stackoverflow going to be?

Instead, I am wondering why there isnt a federated open source search engine. It would be a cloud of nodes, each node spidering and indexing a small hash bucket of urls. With a million such nodes we could have a live updated, distributed search engine to replace Google. We could run millions of queries without paying someone - we'd pay back by serving as a node, just like in BitTorrent. With all the interest in privacy, I wanted to see more discussion of replacing Google with an open, non-censured and private protocol for search.

<comment><USER15> In the same vein, what good is an index of the web going to be? Googles value isn't their index, its how they translate your search terms into good results from that index. Anyone can index the web, not everyone can make sense of it from human questions.

<comment><USER14> There has been a lot of research in search engines since AltaVista collapsed and Google won. Much of it is in public domain.

</comment></comment></comment></comment><comment><USER16> I'm sure DDG works great if you're an American. As someone who has tried a region specific version of DDG though, I can say the results are downright terrible.

Some say the DDG bangs are a solution. What do I win by doing that? It only made me resort to !g all the time, because the results were so bad.

Now I use [https://www.startpage.com/](https://www.startpage.com/) with region set to Swedish. It's practically a proxy for Google search, so it gives me the right results but sans the filter bubble experience (yes, I want the regional bubble). If you're a non US user, I can recommend it.

<comment><USER17> I must say I'm having the reverse experience. I very seldom use its Danish regional setting, but even for Danish searches DDG is fine. It's only 4-5 times a month I need the !g bang. But knowing about Startpage, I now use the !sp bang instead of !g.

Indeed, due to the bangs, I know use DDG as my proxy to a lot of other search engines, like Wikipedia (!w/!wda/!w..) and Wiktionary (!wikt), and even obscure ones like Memory Alpha (!memoryalpha). Very handy indeed.

<comment><USER18> The wikipedia-in-certain-language bangs are very useful, indeed.

</comment></comment></comment><comment><USER19> What's with this person's writing style? I've never seen so many literary tics in one place. "In short", 14 usages. Almost every sentence starts with a prepositional phrase like "In fact...", "Yet...", "However..." Confusing sentence structure like "How would have the net looked like?" It's so irritating to read.

<comment><USER20> E-book writer... should explain his style. Here's a TLDR summary:

\- Google huge, DDG tiny.

\- Gabriel Weinberg serial startups all failed until he sold one for $10MM, which allowed him to do and focus on DDG.

\- Privacy a big deal these days and DDG markets towards that.

\- DDG makes money from keyword advertising and affiliate revenue.

\- You too can profit from something!

<comment><USER21> How much is $10MM? Like, a million million? Or is a millimeter thickness of dollars?

<comment><USER22> No, it's a thousand thousand (roman numeral). In finance, using MM and MMM is quite common (although bn is often preferred over MMM).

<comment><USER20> yeah, I am in finance. At the bank I used to work at we'd use $1M for $1,000 and not allowed to use $1K.

<comment><USER21> That seems like a misunderstanding waiting to happen. $10M is 10,000,000 to me but only 10,000 if you work in finance??

</comment></comment></comment></comment></comment></comment><comment><USER23> The thing with Duckduckgo is the choice has to be ideological or you will do a search again on Google again just to ensure you are not missing out.

This is Google marketing and brand perception at work because Google results of late, 3 years, have been unimpressive and you have to sift through pages of useless links and content to find any relevant information beyond the usual suspects one already knows, so their intensive spyware operations doesn't seem to help search quality.

It's surprising there are not more experimental search projects. One would have expected a steady stream of regular attempts but not a single credible effort exists. There was once an alternative search project called Cuil that just seemed to fizzle off.

<comment><USER24> I have to say, I go to the second page of Google search results _maybe_ once every couple months. I'm not sure what you are searching for but that has not been my experience at all. It seems to always be getting better.

</comment></comment><comment><USER25> The useful insight about DuckDuckGo, which the author misses, is that even with 0.1% market share, a search engine can be profitable. Search ads are valuable, because they appear when someone is actually looking for something and likely to buy. Almost all other ads are merely annoying interruptions.

DuckDuckGo isn't even a full search engine. They don't crawl the whole Web. The heavy lifting is done by Bing and Yandex. That allows DuckDuckGo to have coverage without much infrastructure. That's what makes the business possible without too much expenditure.

<comment><USER26> The article also sort of answers my question how a third party gets access to Bing and Yandex's indexes. I know they offer access, but how is it paid for? By using the founder's money from an exit, and then some VC money until profitable.

</comment></comment><comment><USER27> "If I do the same with Google, you get over 11 billion results!"

Yes i have this often, and then when i click for example from page 5 to page 6 it suddenly says NO RESULTS and im always left flabbergasted with the thought "But google.. you just told me i got 11 million results to search through myself..."

<comment><USER28> I've had the same experience even with the 2nd or 3rd pages. Makes you wonder if google isn't suffering from grade inflation.

</comment></comment><comment><USER29> Might be an interesting article, but that position:fixed footer with ongoing animations is just too atrocious.

<comment><USER30> On mobile it asks for an email in a persistent footer that takes up 1/3 of the screen. When you type a fake email (there is no way to remove it) it brings up another footer asking if you would like to have such a footer on your own website ( also unremovable). Thank God for reader mode.

<comment><USER31> I burst out laughing, thinking they were trolling. But it's a real product.

</comment></comment></comment><comment><USER32> I use DDG everywhere except at work, where (for some reason) it's blocked.

The one silly thing I miss about not having DDG at work: in DDG, I can type "new guid" and it gives me a new random guid. If there's a way to do that in Google, I haven't figured it out. (And yes I know there are a million other ways to get random guids. It's just convenient for me to get them this way.)

<comment><USER33> Hey Andy,

Do you know what firewall system your work uses? We were blocked in a few of them but found that reaching out to those companies got it fixed.

Would be happy to help investigate!

Disclaimer: DDG staff.

</comment></comment><comment><USER34> I tried DDG several times over the years before finally sticking with it about 2 years ago. Now it feels weird to go back to Google because I've gotten used to using some of the bangs.

I love DDG and what they stand for, and I'll gladly trade the creepy personalized results for the more organic results I get there.

</comment><comment><USER35> _How do you make money if you don't track users?

All you need is a keyword. For instance, let's say I'm looking for a new computer, I insert the keyword in the search box "new PC" and all you have to show me are ads related to that. I don't necessarily have to see all the things I've been looking for in the past._

Seems eminently sensible to me. And much more likely to produce ads that are relevant to what I'm looking for _right now._

<comment><USER9> Yeah, I often wonder how what Google is doing can really be so much better that it justifies leaking billions of users' information all over the place and the massive infrastructure costs that they must have.

</comment></comment><comment><USER36> Much ink is being spilled currently over the evils being perpetrated by the tech giants. What's so exciting about DuckDuckGo is

1\. DDG competes directly with Google in Google's core market

2\. Not being evil is DDG's differentiator _for real,_ they're not just paying lip service

3\. DDG is small and profitable... in other words, they're a successful startup.

It can be done. It is being done.

</comment><comment><USER37> I know the American Dream story of somone small being big corporations is appealing, but s/Solopreneur/Team/.

<comment><USER38> "Solopreneur" ...what a stupid, stupid word to make up.

</comment></comment><comment><USER39> I've had more than one family member uninstall DDG thinking it was malware. DuckDuckGo, imo, implies some degree of arbitrariness in whether they'll take your search query seriously. The article mentions he came up with the name before he had a product. It's just too bad he didn't think of a kids game just before this search engine.

It's my default search engine and without fail when a non-tech colleague is over my shoulder for an internet search they bust out laughing at the name and are insistent that it's a prank website. I persist and calmly explain. They relent and give that look you give a crazy person you don't want to argue with.

<comment><USER40> I remember when Google looked that way too. The lack of other links, gif animations and backgrounds, the prototype-style primary colors . . .

<comment><USER39> But the name Google implies a mind-boggling large number of results or websites that it's processing or searching for you. Maybe the name has a degree of playfulness, but it's not named after a children's game that revolves around the arbitrary choice of an unlucky child.

They also had the luxury of entering the market before there was a dominant player. Not to mention the brilliance of the pagerank algorithm.

</comment></comment></comment><comment><USER41> I feel like one of the appeals to using DuckDuckGo is that it simply isn't Google and, therefore, doesn't come along with all the ethical baggage. That's true now, but who's to say that will continue to be true in the future? At the end of the day, the same forces still drive large businesses to cut corners, costs, and creeds if they want to compete effectively with one another.

<comment><USER39> There are no guarantees, but if they're not currently amassing data trails then they won't be an attractive acquisition target for a data aggregator. If they ever change their privacy policy, you just stop using it.

</comment></comment><comment><USER42> I've been using DuckDuckGo for about a year now. I would say it has complete parity with every search engine other than Google, and only seems to miss things when I do very specific searches. A common example is looking up an error in dmesg: I can usually get a specific bug thread on Google, but only more general results on DDG.

</comment><comment><USER43> I've used DDG on and off since the initial HN submission, and I have to agree with the 'like quitting smoking' comment.

In the past I would switch to it when I'm feeling some google-morning-after- shame (e.g. after seeing some targeted ads), I would stick with it for a day or two, but eventually go back to the 'what I wanted is on the first page' magic of google.

I've used DDG more consistently since changing the firefox search default, but there are still some things that I end up googling - sometimes DDG shows too many irrelevant results on the first page.

I think paying more attention to the bangs and moving away from 'keyword' searches will probably help - after all a tool is more useful if you learn how to use it properly - but for some topics (e.g. Haskell examples) if the first answer it finds isn't what I was looking for, the next couple of pages of results are usually useless too.

</comment><comment><USER44> Off-topic:

Can we _please_ stop with this trend of random websites asking to spam me with notifications? Is there some framework everyone's using that's trying to grab notification permissions at every possible opportunity? At the very least, sites should at least tell me what they're going to have pop up as a notification and give me the option to subscribe to those notifications through some link in the actual site (like what Gmail does when desktop notifications aren't enabled yet).

I understand notifications are useful for "webapps", but for a blog post they seem entirely useless, so every time I see the prompt to allow notifications I reject it immediately on the basis of it almost certainly being spam.

</comment><comment><USER45> Having swapped over to DDG a couple years back, I recall my frustration and Google-fallback habit lasting for ~6mo. At this point I'll end up using the big G maybe every week or two--mostly I forget that they're there.

Honestly, my second most-used "search" style interface these days is Wolfram Alpha. Google comes in _third._

Say what you will about their respective search result quality, I found that a lot of the delta between DDG and G! could be made up by just spending time with DDG. I believe I've subconsciously adjusted both my keyword structure (I'm conscious of using "explicitly" "quoted" "keywords" to ensure they're included in my results) and my expectations.

</comment><comment><USER46> But how good can a search engine be if it _does not_ track its humans, and therefore _cannot_ use browsing / search history as contextual clues for query ranking / semantics ?

It might be very good. But can such a system compete with one that does use search history as context?

<comment><USER47> There used to be a concept of google-fu, where writing search queries was a skill in itself. I got quite good at finding what's needed. Then, over time, search results became less and less useful. I noticed that search results seemed to be weighed more on my history than the query written. I'd get results with words synonymous to web page titles I'd recently visited, but otherwise no connection to my search. To get decent results now I have to use incognito or change the search settings to be "verbatim." I hoped things would get better, but years later things are the same.

DDG gives more predictable results. It reminds me of why I started using Google over other search engines decades ago.

</comment></comment><comment><USER48> I really, really want to like DuckDuckGo. But every single time I've switched from Google, I've found them to be much worse at returning the best results for what I'm looking for.

For example, let's say I want to look up "ear infection". Google will spit out a bunch of info right on their results page, often saving me the trouble of even going to another site. DDG however will just give me 10 webmd links.

If they want to compete, the privacy angle isn't enough. They need comparable functionality as well.

<comment><USER49> Try startpage.com. I moved away from google a year ago and never looked back. I've never _once_ needed to fall back to google. That's probably because startpage.com is basically anonymized google.

I know we're supposed to root for the entrepreneurs here on HN, but DDG honestly seems like more hype / marketing than a decent product. You never hear anything about startpage.com (maybe because it has a boring name?) but it just works.

<comment><USER48> Thanks for the suggestion. I just gave them a try with my "ear infection" search, and found that while they provide similar links, they don't give the additional info (like the summary of what it is and the "People also ask" dropdown box) right on the results page that google search does. This is especially important to me on mobile, especially when I have a weak cell signal.

</comment></comment></comment><comment><USER50> DuckDuckGo is OK but they seem to be penalising single-page apps so my project websites rank low on DuckDuckGo but high on Google. So I can't support DuckDuckGo.

<comment><USER51> Some might see this as a win for DuckDuckGo

<comment><USER52> People who like inaccurate search results?

<comment><USER53> That seems to be pretty one-sided view on decisions one makes out of trade- offs. Having iPhone 4 few weeks ago with js obviously off and now having SE with js:on, I'm not sure whether my web experience got any better or not at all (overall phone experience got better, no doubt).

</comment></comment></comment></comment><comment><USER54> I used to use ddg, then startpage, but I've been using a self-hosted searx for a while now. It's a meta-search engine that scrapes other sites. Results are always great, and it's nice being able to enable specific engines for different sites. Check it out: [https://searx.me](https://searx.me)

<comment><USER55> That one seems down at the moment, at least for me. List of instances:

[http://stats.searx.oe5tpo.com/](http://stats.searx.oe5tpo.com/)

</comment></comment><comment><USER56> DuckDuckGo has really, really improved on their results for me who do not live in USA. Before, it was pretty much unusable. But today I use DDG as my main search engine on all my devices.

You should too.

<comment><USER33> Our engineers have been working really hard on international results - I'm sure they will be happy to hear it's making a difference for our users!

Disclaimer: DDG staff.

<comment><USER56> Glad to hear that! Big difference and I am glad you're not completely US- focused which is unfortunately common. :)

</comment></comment></comment><comment><USER57> I still don't understand - does DDG have its own crawler and index, or is it outsourcing all that to third parties?

If the second, as I suspect, it's not really long-term viable anyway, is it.

<comment><USER58> Second, they might crawl a few things, what percentage of the queries are answered by their index, that information they will probably not say.

</comment></comment><comment><USER59> Isn't DDG basically Bing + bangs? Doesn't seem fair to credit it at beating Google without highlighting the billions of dollars that MSFT has invested.

<comment><USER33> DuckDuckGo uses multiple sources for our results, as well as our instant answers and bangs.

But the most important feature is that DuckDuckGo is the search engine that doesn't track you. We don't store your personal information and we don't sell you out to advertisers.

Our goal is to raise the standard of trust online, and so at DuckDuckGo you are not the product, search is. We think that's an important difference.

Disclaimer: DDG staff.

<comment><USER60> _" We don't store your personal information and we don't sell you out to advertisers."_

But is that information collected by you or anyone affiliated with you or given to anyone?

How about information that doesn't directly identify you, but could be used to identify you? (things like browser fingerprints, IPs, cookies, etc)

<comment><USER33> If you want to look at the nitty gritty we have a very simple and straight forward privacy policy:

[https://duckduckgo.com/privacy](https://duckduckgo.com/privacy)

We do not collect or share any information that can be used to personally identify you.

</comment></comment></comment></comment><comment><USER61> I see DuckDuckGo as an improved search bar more than as a search engine, though it's admirably fast at the latter.

DDG's bangs are effectively a CLI interface to web searches. !g does google, !yt does YouTube, !w does Wikipedia, !wa does Wolfram Alpha, !scholar does Google Scholar...

Huge timesaver.

<comment><USER62> It's interesting, since browsers have had custom keyword searches for ages, and I've known that and just never taken the time to set them up.

Having DDG with its bangs I use them all the time, and if there's some specific new site that I want to search there's (so far) always been a fairly guessable bang already set up for it.

I find it exceedingly convenient.

</comment></comment><comment><USER63> Isn't DDG just using Yahoo on the backend to perform the search (which in turn is using Bing)?

What prevents Yahoo from (a) offering a competitive search to DDG or (b) Yahoo from shutting of DDG from using its search engine?

<comment><USER64> DDG is using over 400 sources apparently, of which one is Yahoo [https://duck.co/help/results/sources](https://duck.co/help/results/sources)

<comment><USER65> > [https://duck.co/help/results/sources](https://duck.co/help/results/sources) > "We also of course have more traditional links in the search results, which > we source from Bing, Yahoo, and Yandex."

I thought they where re-implementing search.. Are they "just" aggregating?

<comment><USER66> I recently spoke with Gabriel and his view was that they're going to continue to outsource / proxy / aggregate unless they need to implement their own search. Right now it's not worth the investment apparently.

<comment><USER52> Yeah because Search is an actual hard problem and putting up a proxy like DDG is not. DDG will never have any real value beyond that.

<comment><USER18> Eliminating tracking of your searches is not valuable?

<comment><USER52> I said value beyond that. That meaning the privacy proxy.

</comment></comment></comment></comment></comment></comment></comment><comment><USER67> I created a website a year ago and I simply cannot find it using Google. Searching for the site title: Google - not in top 100, DDG - result 8. I am in Google, since I can search for the exact URL (which is the same as the site name), but otherwise I don't show up. It's incredibly frustrating considering the amount of no-content and repeated articles Google does return.

Related, my biggest gripe with the WWW today is how difficult it is to find personal websites among (what I'd call) all the ad-revenue garbage.

<comment><USER68> Just to be thorough; did you try this: [https://support.google.com/webmasters/answer/6259634?hl=en](https://support.google.com/webmasters/answer/6259634?hl=en)

It is an extra step to make google aware of you.

<comment><USER67> I did, so the site is in Google and is crawled by Google (Analytics I believe has the option to do that). I also went through various SEO suggestions, set up Google Analytics, added meta tags, improved load speed (it's a pretty minimal static page anyway), among other things.

</comment></comment></comment><comment><USER60> When I search for:

      "strace" "hex" "ascii" site:stackoverflow.com

on DDG lite:

      https://duckduckgo.com/lite/?q="strace" "hex" "ascii" site:stackoverflow.com&kd=-1

the returned results only show "hex" and "ascii" but do not include "strace". I expect results with all three search terms (and if that fails, to return nothing).

</comment><comment><USER69> I tried to find some info on the architecture of DDG, and found this article from 2013: [1]. It seems they used Solr at the time.

Is there anything more recent?

[1] [http://highscalability.com/blog/2013/1/28/duckduckgo- archite...](http://highscalability.com/blog/2013/1/28/duckduckgo- architecture-1-million-deep-searches-a-day-and-gr.html)

<comment><USER70> If you reverse some of their servers, you'll also find they're hosted mostly on AWS. So that privacy thing....

</comment></comment><comment><USER71> Interestingly, contrary to most comments, i recently had to switch to Google after using DDG for about two years. It was just getting slower and slower - across all devices. Yes, programming and technical questions are answered better than Google, but rest of the queries were returning lesser optimal results - most likely it has to do with privacy and data collection.

</comment><comment><USER72> I'm a fan of DuckDuckGo and I've found myself using it more and more these days. I actually had to change my launcher on Android because the default Google Go launcher that comes packed with the Moto X Pure has no option to remove the Google search from every home page. I switched over to Nova launcher just so I could put a DDG search bar in its place.

</comment><comment><USER12> I've been using DDG for 3-4 years now. Good results generally, love the bangs.

Still thin results an a few sites, Ello and Reddit comments particularly IME (I use and search both frequently).

For console, there's [https://duckduckgo.com/lite](https://duckduckgo.com/lite) (no JS).

Hit counts and date-bounded search stills sends me to Brand G.

</comment><comment><USER73> I'd like to act as devil's advocate for a bit.

Showing ads according to a search term is totally possible, but having no attribution attached to a "click" or to an "impression" give very little advantage for the marketer who is paying for the ads. There are cost models allowing to pay for the actual action (like installing an advertised application) not just for clicks or views. Re-targeting people who had expressed their interest in a product is a useful tool for marketers as well. Having some kind of link back to the advertising campaign, which your users came from along with their LTV allow you to measure campaign productivity, which helps optimize future campaigns. And much more.

I really like the idea of not being tracked on the Internet, but it's seems like currently it's not feasible to remove a tool many marketers get used to.

<comment><USER73> People, who downvoted my comment - care to explain? Does downvote button on HN compare to "I don't like what you're saying"?

</comment></comment><comment><USER74> Does anyone know of a way to help DDG improving by passing them statistics about how I use !g as well as passing information about the results I prefer from google? I can only assume it would be riddled with privacy issues but would be happy to occasionally provide it.

<comment><USER33> Whenever you do a search you'll notice there is a "Feedback" button in the top right-hand corner of the screen.

This allows you to lets you share your feedback about the site or specific results with us. Your personal information is not recorded.

If a query isn't returning as expected you can share it there!

Disclaimer: DDG staff.

<comment><USER74> Thanks! I'll keep this in mind.

</comment></comment></comment><comment><USER75> Bing, Seznam, Yandex, Baidu.. there are many other search engines with surprisingly good results. Give it try.

I started using alternatives, for privacy, but also because they are often better. Google Maps can not even find near pharmacy in EU country.

<comment><USER76> You think Bing, Yandex, and especially Baidu are better for your privacy than Google?

<comment><USER77> With regard to Yandex and Baidu: there's the notion of threat proximity. Google is a US company, beholden to US laws. If you reside in the United States, you also reside in the jurisdiction of state actors with the capability to _get at you_.

If Putin himself showed up to Yandex HQ and demanded a full data dump of every search you made, he is still _incapable of touching you_ , whereas J. Random Agent at the FBI can throw you in a 6x8 steel box.

<comment><USER76> Has anyone ever been sent to jail in the U.S. because of their (innocuous) searches?

</comment></comment></comment></comment><comment><USER78> I've been using it for ~ 4+ years and it's great most of the time. I sometimes jump to Google if I can't find what I'm looking for, but these excursions generally aren't very successful.

</comment><comment><USER79> Insofar as DDG is a mashup of many search engines, doesn't it technically have a _massive_ team? If you add up the teams building the products that feed DDG, then you probably end up with a much larger team than the one building Google Search.

I might describe that as gaining on Google, but still losing quite badly.

I'm still impressed by DDG of course and welcome some alternative to Google, which is a dangerous monopoly with a search that I've come to despise in recent years due to its ignoring what I've actually typed.

</comment><comment><USER80> DDG is really great! I use it on all my devices since January. I've tried it before to by changing the default to DDG on my iOS devices. However, I've always switched back to google after a couple of weeks. Not this time.

Every now and then when I'm not happy with the search results, I use the "g!" Shortcut to get the google results, but the occasions I'm doing this is definitely decreasing.

</comment><comment><USER81> OT:

> _" Google would be an elephant while DuckDuckGo is a mosquito (this is not > to emphasize; I'm actually making things better for DuckDuckGo)."_

Actually the Author is not making it better, in fact worse. From the numbers in the OP it appears Google is about 1000 times bigger (roughly) than DDG.

A better, and perhaps more apt comparison would to compare Google's Elephant to DDG's Badger (small badger).

</comment><comment><USER82> Is there going to be a replacement for DuckDuckHack[0]?

Had a cool instant answer integration I wanted to contribute, that I hope would be useful to a lot of people (primarily students/scientists).

Seems a real shame for there to be no route to community contributions in the future.

[0] [https://duckduckhack.com](https://duckduckhack.com)

</comment><comment><USER83> One reason why Google seems to do better is because it knows more about me. For example when I search for "reverse a LL" shows me reversing linked list on Google (which is what I intended) but something completely else on DDG. Maybe because Google just knows that I tend to search for programming stuff.

</comment><comment><USER84> I switched to duck duck go recently and for me it is good enough, when I feel I get bad results or not what I need I search with !g and for my surprise google returns same results most of the time. It would be good to have more duck duck go like sites/projects in case ddg fails or goes evil.

</comment><comment><USER85> I've been using DDG as my primary search for about 2 years now, it rarely ever falls short for me. When it does, it's usually related to some type of image search I'm doing, I think google still has nicer tools for that.

Overall, super happy with DDG.

<comment><USER86> Agreed. It's not perfect but I'd prefer to give it the first chance(s), and if necessary fallback to Google, etc.

</comment></comment><comment><USER87> I always use DDG first, and then as a last resort ill look on Google. I think I only really checked on Google maybe 2 or 3 times this past month. For me Google is mostly irrelevant, and I could quite happily not use it.

</comment><comment><USER88> Persoanlly I love DDG. It handles a vast majority of my search traffic quite fine.

Quick tip: Rather than falling back to Google, try the "!sp" bang command for StartPage, which crawls Google to supplement it's results.

</comment><comment><USER89> I've switched to duckduckgo for security reasons and noticed not much difference. Even when I switched back to google for something I couldn't find with duckduckgo the search results are very close.

</comment><comment><USER90> I'm curious how DDG's affiliate revenue compares to search-based ads. The article glosses over affiliate programs, but they are an interesting revenue alternative to ads.

</comment><comment><USER91> I don't understand what's the difference between Google and DuckDuckGo when all of my traffic is going through Comcast and AT&T anyway. There's no privacy.

<comment><USER38> Unless AT&T/Comcast are somehow MITMing your SSL connections (unlikely), then your search terms are all encrypted and all the ISPs know is that you are using DuckDuckGo.

</comment></comment><comment><USER92> According to uMatrix, DuckDuckGo serves 43 cookies from Google? Does it only "No Follow You" if you're proactive enough block them using uMatrix?

<comment><USER93> Privacy badger detects zero trackers. When I open my developer toolbar, all hits when I search are to duckduckgo.com. When I installed uMatrix, the only thing I see are references to duckduckgo. And _zero_ cookies. This is using Firefox on Linux.

It appears as if Google has infected your internet.

</comment></comment><comment><USER94> Two years and a half just with DDG. It's really ok for me. Just a number of !g sometimes. Google searches are surely better but I feel comfortable with DDG.

</comment><comment><USER95> Love how everybody concerned about their privacy are most likely querying DNS via 220.127.116.11.

But I understand, I try to think of the internet as a non-privacy place.

<comment><USER96> What a strange straw man. People who are very concerned about their online privacy are likely not using that as their DNS.

</comment></comment><comment><USER97> DuckDuckGo is great.

Also, the HBS case study prose style is annoying enough in itself. To have the style emulated by a non-native writer is fun. And weird.

</comment><comment><USER98> But doesn't DDG just repackage results from Yahoo (now Bing)? At least, it was doing that a few years ago when I last checked.

<comment><USER99> They also have their own crawler: [https://duckduckgo.com/duckduckbot](https://duckduckgo.com/duckduckbot)

<comment><USER98> But that doesn't mean they don't just repackage Bing's results (with some of their own thrown in).

</comment></comment></comment><comment><USER100> Couldn't access the website - too many pop-up! Hopefully the HN comments made me want to try DDG again

</comment><comment><USER101> I'd just like to say that this is a great article--stuff like this is why I keep coming back to HN.

</comment><comment><USER102> Why do search engines have to exist on a website? Couldn't they be a built-in part of the browser?

<comment><USER103> Not really. A search engine crawls the web (billions and billions of sites) and indexes them for quick and easy access. If you use Outlook for email, it's kind of analogous to how the search option works there. When an email arrives on your company's system, the mail server processes the mail and indexes it for searchability. The key here is that either your computer or your mail server have to have a copy to make it searchable.

So, for the case of web searching, you would have to know/read the content on the entire web. This is the "crawling" process that search engines perform. They read it and store/index what they need for searchability. It is entirely infeasible, obviously, for anyone (let alone every person) to do so on their individual devices.

<comment><USER102> Thanks for the answer! Couldn't Apple crawl the web and store the index for safari to search through? This way, an iPhone user would search Apple's index instead of Bing's, no website needed.

<comment><USER103> If I understand your question, the answer is that what you describe is basically the same thing currently on Android phones with the Google search widget thing (I think it's called Google Now).

It can perform a search without going to Google's main search page. It may or may not open the results in a browser window, though (I don't know because I don't use it). But regardless, it goes through Google's main search system, much like the proposal you made for Apple.

Whether or not it is accessed through the site directly is kind of irrelevant. It's still accessing the same remote system in the same way. Whatever interface you choose to use is just a user preference, really. You're still submitting a query to the site/service over an internet connection and you are getting the response back with the results.

Does that answer your question? I'm not sure if you were asking if this hypothetical from Apple could avoid using Google or if you were asking something else. Because if you are trying to avoid giving your data to Google, then you should also be concerned with Apple.

<comment><USER102> Excellent explanation!

>I'm not sure if you were asking if this hypothetical from Apple could avoid using Google

Yes, that is what i was trying to get at!

>Because if you are trying to avoid giving your data to Google, then you should also be concerned with Apple.

Definitely!

>Whether or not it is accessed through the site directly is kind of irrelevant. It's still accessing the same remote system in the same way.

I'm just curious if there's a business opportunity for Apple here. They could basically make mobile safari switch to the hypothetical "Apple Search" for queries made in Safari's address bar (I personally don't search from google.com). It would of course only be worth anything to the end user if they saved time not being routed through google.com

In a similar vein, I've read some comments describing "Siri" as a way to get a bite of the web-search-cake.

<comment><USER103> Yeah I think there is a business opportunity for them if they really wanted to give it a go. DuckDuckGo leverages Bing, Yahoo, and a number of other sites. I don't see why Apple can't do something similar and forge similar agreements and then use iOS, Siri, and everything else they have as part of their "ecosystem" to drive searches through their new platform.

</comment></comment></comment></comment></comment></comment><comment><USER104> DDG user. I use this to cook pasta: 7 min !timer (goes off to google & starts a timer)

</comment><comment><USER105> Use search engine[1] that respects[2] your privacy

[1] [https://www.ixquick.eu/](https://www.ixquick.eu/) [2] [https://www.ixquick.eu/eng/privacy- policy.html#hmb](https://www.ixquick.eu/eng/privacy-policy.html#hmb)

<comment><USER106> The results are terrible. I tried a simple search for a large Porsche dealership which is down the street from my house and it did not even come up in the top 5 results.

Would be a great search engine if it was the year 2001.

<comment><USER105> Well, for such use cases Google is a better tool indeed. I personally know my neighborhood and never google what is down the street. And if I need dealership I will try to find it on car maker website which is easy to look up even using search engines from 2001.

</comment></comment></comment><comment><USER107> I'd love me some DDG Scholar functionality (not the Google Scholar bang)

</comment><comment><USER108> They're beating Google at monopolizing the internet's advertising?

</comment><comment><USER109> How do they plan to scale with the number of users?

</comment><comment><USER110> Using Google to analyze its own competitor seems a bit questionable. Though with the difference in usage being what it is, I doubt Google would bother with any funny business.

</comment><comment><USER111> DDG is just icing on cakes, a convenient UI to search the web/search engines. Now trend is shifting towards chatbot, is DDG still relevant?

</comment><comment><USER112> i use ddg for finding things that haven't been taken down by dmca notices.

</comment><comment><USER113> "Google would be an elephant while DuckDuckGo is a mosquito (this is not to emphasize; I'm actually making things better for DuckDuckGo)."

Why would this guy go out of his way to say this metaphor is actually literally informative and generous towards DuckDuckGo while apparently having exerted no mental effort and being incorrect by orders of magnitude on his own data?

If you're most generous to the writer, you get 12 million hits /day vs 13 billion hits /day according to his data from Wolfram, for a ratio of ~1/1100, which applied to a pygmy elephant of 5500 lbs yields a corresponding weight of ~5 lbs for the mosquito, or over 2,000,000 mg, versus the average mosquito weight of about 5 mg. Even if he meant a small 2000 kilogram pygmy elephant and a 20mg elephant mosquito, he's still 5 orders of magnitude too generous towards Google in mass comparison in his metaphor in which he is "actually making things better for DuckDuckGo".

I think most people think of animals like African Elephants when they hear "elephant", though, in which case you're looking at a 13,000 pound animal versus a ~12 pound one if you use hits as your metric, or a ~95 lb one if you go by visits as your metric.

So an actually fair metaphor is if Google's an elephant, DuckDuckGo is somewhere between a goose and a hyena. Better watch out, Google.

<comment><USER114> That was one of the most HN comments I've ever read.

Hey, it's just a metaphor.

Scales and ratios not needed.

edit: Wasn't meant to be negative ... as I could see myself writing something like that and my GF giggling at me for a week ...

<comment><USER115> I agree with parent, it's not just a metaphor but a wildly inaccurate guess that makes the reader think duckduckgo is more irrelevant than it really is.

> _That was one of the most HN comments I 've ever read._

What does this mean?

<comment><USER116> HN has a reputation among some[who?][weasel words] as a place where many commenters are some combination of pedantic, overly literal and out of touch with mainstream culture. Taking this metaphor to pieces would seem to check the first 2 boxes pretty neatly.

<comment><USER117> You actually make it sound bad. I would more likely to say they are precise and usually won't allow for inaccurate information to fly through the front page without pointing out the issues.

<comment><USER118> See guys, this is why we don't get invited to parties...

<comment><USER119> We'll just have to hold our own shindig, with educational flash cards and nature documentaries.

</comment></comment></comment></comment></comment></comment></comment><comment><USER120> duck.com is owned by Google.

</comment><comment><USER70> Does anyone remember how back in the day, we had Lycos, Hotbot, Yahoo, Dogpile .. a whole host of search engines. You'd go to two or three to get an idea of what was out there. People had different indexes and there were a lot of players.

Today, it's just Google. I've been using DDG for a few years, but about 1/3 of the time I add an !g because I don't find the results I need on DDG.

The cost of entry to the search market is exceedingly high right now. This is a pretty good article of detailing how one person was able to come up with an idea and challenge the behemoth in very niche areas (privacy/the nsa leaks were probably the reason I started looking at/using it around 2013).

Yet I still miss the days of using multiple search engines; seeing a variety of results. I hate the de factor standard of Google. When a company controls that much of search, they get to define the narrative. They literally shape the way many people perceive the world.

I wonder if tech will get to the point where indexing will be easier and we'll see more solutions that are cheaper and that can crawl larger datasets with lower processing requirements. Maybe the next step will be distributed search with shared indexes?

In any case, Google can't remain on top forever (at least I hope not). It'd be nice to see more tech in this space, but it's an incredibly difficult problem. There is reason Google climbed to the top like it did.

<comment><USER121> I really like the localized index idea someone posted earlier. If I could get 60% of my searches answered locally for the cost of 100 gigs of space I would use it. Even more so if I could host it on a local server somehow.

<comment><USER21> Why though? Is your connection that bad? I wouldn't want to use 100GB of precious SSD storage for something I can get instantly in the browser

</comment></comment></comment><comment><USER122> Why does DDG not present "direct" URLs in its search results?

Why does DDG, like Google, _by default_ instead prefix all results with a URL pointing to DDG servers enabling DDG to track what results that the user clicks on?

Google started doing this some years ago, e.g., all the URLs in search results are prefixed with something like [https://www.google.com/url?q=](https://www.google.com/url?q=)

Needless to say, this serves no useful purpose for users and it showed the direction the company was moving in.

Easy for any nerd to remove client-side, but it is on by default and this will catch many non-technical users who do not know about them or how to remove them.

In DDG the prefix is something like [https://duckduckgo.com/l/?uddg=](https://duckduckgo.com/l/?uddg=)

Even assuming no logs are kept with client IPs, and no correlation can be made by any party in possession of DDG's logs bewteen client IP and clicked URLs, this practice is still collecting data. The data collected is not merely _what searches a user submits_ i.e. search data, but also data about the user's _browsing_ , i.e., which URLs she chooses to follow. And of course it collecting this data _without asking for the user 's permission_.

If the company can argue having this data is useful to the company and therefore somehow useful to users because company will make better website/software (a common argument made by many web companies caught collecting user data by default), then it seems the respectful thing to do is ask users if they want to contribute such data. Let users make the choice.

As I recall this default prefixing of results is not something that search engines before Google used to do. Nor did the original Google do this either. It is something that Google started doing some years ago.

Note: Some browsers allow the user, e.g., to preset a default HTTP referrer, to set it to the target URL, to send an empty one, or to not send this header at all. Same options as offered by "meta referrer" except it is _controlled by the user_ in their _browser_ , not via a third party website. I have no idea if the popular browsers have such settings.

Finally, a problem with having bad design choices (such as prefixing URLs) and then offering users the option of changing the "settings" _via a website_ is that this usually requires Javascript or cookies, because as everyone knows HTTP was designed to be stateless. Cookies and Javascript are two things privacy conscious users may want to avoid. By default DDG does not set cookies or require Javascript. (Good.) But if in order to change a bad default, the user has to enable cookies or Javascript, then we have sacraficed the goal of no cookies or Javascript required. (Bad.)

<comment><USER123> > Why does DDG not present "direct" URLs in its search results?

They explain it here: [https://duck.co/help/results/rduckduckgocom](https://duck.co/help/results/rduckduckgocom) It's to prevent sites from seeing what search term you found them by.

Turning it off: Menu button at the top right of search results -> other settings -> Privacy Tab -> Redirect (Toggle)

Edit: Note that I have the redirect setting turned on, and don't get the redirects. Presumably because I'm running firefox nightly and they know that this browser supports rel="noopener" (which they use on the link).

</comment></comment><comment><USER124> I am thinking about the current debt based economy which is based on growth and stock prices, and Google and the tracking ad bubble is a good example of the problems. Brendan Eich talked about it: [https://twitter.com/BrendanEich/status/870483249587519489](https://twitter.com/BrendanEich/status/870483249587519489) Google was VC funded so had to exit using IPO (in this case).

</comment><comment><USER125> I've switched over on my phone, mostly because I'm sick of not having any other way to opt out of AMP.

</comment><comment><USER126> I would actually switch to ddg if they actually bothered to free more parts of their site and backend. Right now their JS is nonfree for example.

</comment><comment><USER127> Is there any practical decentralized alternative search engine? YaCy seems dead.

</comment><comment><USER128> Hell yeah duckduckgo owns

</comment><comment><USER129> Why not using Yandex directly? I mean, really?

</comment><comment><USER130> DuckDuckGo is really an amazingly stupid name for a browser brand, and the logo completes it.. How small would Google be with such a name?

</comment><comment><USER107> DuckDuckGo, powered by Yandex? In all seriousness, does that mean Putin and Russia would have my searches? Not trolling,just interested to know what I would be choosing by going with DDG.

<comment><USER107> You can negatively vote the question, but the question was in earnest, and I'd appreciate an earnest answer of why their partnership and reliance on Yandex is not a privacy concern.

<comment><USER107> e.g., [https://www.theregister.co.uk/2017/05/30/yandexs_ukraine_off...](https://www.theregister.co.uk/2017/05/30/yandexs_ukraine_offices_raided_for_treason/)

</comment></comment></comment><comment><USER131> I will stick with Google for now. Google usually returns the result I wanted. Google's page is also cleaner.

But when DDG improve the search result, I will make a switch there.

</comment><comment><USER132> The comments on this article mostly talk about the privacy benefits DDG brings. Yet, we've known since the Orange Book (TCSEC) that system integrity is a prerequisite for confidentiality/privacy. As in, the DDG servers and network need some strong security to ensure they're doing what the staff and users think they're doing esp with all the press it gets. Example attacks:

1\. Root the search servers. Gradually leak out what people were doing in a way that looks like fake search results to attackers running "searches" that are actually commands to trigger leaks. The leaked data would be stored in memory temporarily.

2\. Subvert the engine to send malicious JavaScript to users that leaks their search results to a specific location. Might reduce risk of detection by first determining browser configuration for common ways of spotting leaks. Then, don't send anything to _those_ users.

I'd guess the boxes and setups were originally optimized for speed and cost rather than isolation. So, what's security like at DDG? A quick glance shows they run...

"DuckDuckGo is coded in Perl and JavaScript with the help of the YUI Library, served via nginx, FastCGI and memcached, running on FreeBSD and Ubuntu via daemontools. We both run our own servers and have servers on Amazon EC2 across the world."

Probably not that private once hackers are involved with Perl and Ubuntu. On nation-state level with EC2. The servers and cache at least get security updates regularly. So, safe assumption is private against passive collection and low-to-medium-strength attackers.

<comment><USER133> I imagine most people using DDG over Google for privacy reasons are concerned first with what _Google_ does with the data it collects on them and only secondarily with the potential for other actors to get access to that data.

<comment><USER132> Entirely true. I bring it up since the article mentioned the Snowden leaks and data ending up in government's hands. The leaks, esp Core Secrets, also said they had a way to get the FBI to "compel" the backdooring. Others said TAO's hackers often made up for what passive collection couldn't do. If they're in the threat model, the DDG isn't likely the solution in and of itself unless their security has the same threat model.

Good for other threat models that majority of users are actually worried about, though. I did say that in original comment.

<comment><USER133> It's still better than most alternatives in other respects it seems to me. The fact that they go out of their way not to track IPs or store personally identifying information means there is less data available for a hacker to access if their security is compromised. Sure it's possible that a sophisticated attacker could add tracking of information that DDG do not intend to track but the situation is a lot better than with Google. They are also safer from other types of non hacking attacks like court orders by deliberately not tracking information that might be demanded.

Security can always be improved but they already seem to be a much better choice than most of the alternatives.

</comment></comment></comment></comment><comment><USER134> Write a whole article about DuckDuckGo, and don't mention bang searches. !py searches the Python documentation, !w searches Wikipedia, !g searches Google, !a searches Amazon, and better yet, !gde, !wde, and !ade search the German Google, Wikipedia, and Amazon.

For natural language queries, I use !g, because Google is better at this sort of thing. For local queries I use !gde, because Google has better map integration. For most everything else, I prefer DDG, because it doesn't "did you mean DickDickGo?", and it doesn't "I don't know what you're looking for, but here are five ads instead".

<comment><USER135> > Write a whole article about DuckDuckGo, and don't mention bang searches.

Ummm... yes it does. It's literally the very first point the author discusses once he starts comparing the search results. Here's a link to the relevant subsection: [https://fourweekmba.com/duckduckgo-vs- google/#Search_DuckDuc...](https://fourweekmba.com/duckduckgo-vs- google/#Search_DuckDuckGo_vs_Google)</comment></comment>


Mango, Android and Apple Mobile Platform Anthropomorphization (Webcomic) - <AUTHOR> http://toblender.com/dating-mobile-development-platforms/

<comment><AUTHOR> Yeah ironically the apple reporting site was delayed again. Not that it mattered because I didn't end up making sales :P

</comment><comment><USER2> Awesome stuff!</comment>


TXR Lisp - <AUTHOR> https://www.nongnu.org/txr/txr-lisp.html <comment><USER2> TXR is not just a pragmatic, low-dependency, small-footprint tool for hackers, but a bit of a Lisp research platform.

For instance, TXR Lisp doesn't have keyword parameters "natively". But via the parameter list macro mechanism[1], a function can take keyword params, if the symbol :key appears as a left item in its parameter list. That triggers the :key parameter macro.

TXR Lisp also has a _defset_ , very similar to CL's _defsetf_. That provides one of the ways by which we can make a form into an assignable place.

Now, here is the cool part. _defset_ [2] and the _:key_ [3] parameter macro do not know anything about each other. Yet if you write a _defset_ for a form that takes key arguments, it works!

Quick demo:

      1> (defun f (:key x -- key1 key2) (list x key1 key2))
      f
      2> (defset f (:key x -- key1 key2) val ^(f-set ,x ,key1 ,key2 ,val))
      f
      3> (expand '(set (f 1 :key1 2 :key2 3) 42))
      (f-set 1 2 3 42)

By contract, the Common Lisp implementations of _defsetf_ I've looked at deal with keyword parameters explicitly. They analyze the parameter list and know which symbols need gensyms and such.

More complex call:

      4> (expand '(set (f 1 :key1 (a b) :key2 (c d)) 42))
      (let ((#:g0065 (a b))
            (#:g0066 (c d)))
        (f-set 1 #:g0065
         #:g0066 42))

_defset_ ferreted out, through the _:key_ expander, without knowing anything about it, that argument (a b) corresponds to the key1 parameter, and that it needs a gensym, which is then inserted in place of the ,key1 unquote in the ^(f-set ...) backquote template.

[1] [https://www.nongnu.org/txr/txr- manpage.html#N-00B4065C](https://www.nongnu.org/txr/txr- manpage.html#N-00B4065C)

[2] [http://www.kylheku.com/cgit/txr/tree/share/txr/stdlib/keypar...](http://www.kylheku.com/cgit/txr/tree/share/txr/stdlib/keyparams.tl)

[3] [http://www.kylheku.com/cgit/txr/tree/share/txr/stdlib/defset...](http://www.kylheku.com/cgit/txr/tree/share/txr/stdlib/defset.tl)

</comment><comment><AUTHOR> kazinator provided examples in two recent k threads:

[https://news.ycombinator.com/item?id=22631649](https://news.ycombinator.com/item?id=22631649)

[https://news.ycombinator.com/item?id=22666539](https://news.ycombinator.com/item?id=22666539)

<comment><USER3> Thanks. I'd love to see some polyglot comparisons with TXR and other languages commonly used for data processing like Perl and Python.

<comment><USER2> [https://rosettacode.org](https://rosettacode.org) features numerous tasks solved in numerous languages. Some 136 Rosetta tasks are solved in TXR.

<comment><USER4> Link to save you some searching:

[https://www.nongnu.org/txr/rosetta- solutions.html](https://www.nongnu.org/txr/rosetta-solutions.html)

<comment><USER2> That page is prepared using two TXR programs, which happen to be checked in side-by-side:

Suck down the data, print it in a condensed format on standard output:

[https://www.nongnu.org/txr/fetch-txr- solutions.txr](https://www.nongnu.org/txr/fetch-txr-solutions.txr)

Generate pages with frames from above data:

[https://www.nongnu.org/txr/make-pages.txr](https://www.nongnu.org/txr/make- pages.txr)

The workflow is to fetch the solutions into a new file and then to use a text merging tool like meld to update the stable copy.

</comment></comment></comment></comment></comment><comment><USER5> I like it but I don't know what to use it for.

My Lisp usage can be summarized as such:

For anything heavy/performant, I use Common Lisp.

For everything else, I use Emacs Lisp.

</comment><comment><USER2> TXR's FFI can describe a C linked list and convert it in both directions in a call, with correct malloc/free memory management.

Given this code, which is compiled as part of a larger library called "crazyffi.so":

      struct lnode {
        char *datum;
        struct lnode *next;
      };

      void list_update(struct lnode *list)
      {
        struct lnode *iter;
        int i;

        for (i = 0, iter = list; ; iter = iter->next, i++) {
          char buf[256];
          printf("lnode[%d]->datum = %s\n", i, iter->datum);

          /* Edit every node by adding numeric prefix to the string.
           * We free the old datum, and install a newly malloced
           * string in its place.
           */
          snprintf(buf, sizeof buf, "%d:%s", i, iter->datum);
          free(iter->datum);
          iter->datum = strdup(buf);

          /* When visiting the last node, add one more node.
           */
          if (!iter->next) {
            snprintf(buf, sizeof buf, "%d:%s", i + 1, "cow!");
            iter->next = malloc(sizeof *iter->next);
            iter->next->datum = strdup(buf);
            iter->next->next = 0;
            break;
          }
        }
      }

This TXR Lisp code calls the function:

      (typedef lnode (struct lnode
                       (datum str)
                       (next (ptr (struct lnode)))))

      (with-dyn-lib "./crazyffi.so"
        (deffi list-update "list_update" void ((ptr lnode))))

      (let ((ll #S(lnode datum "how"
                         next #S(lnode datum "now"
                                        next #S(lnode datum "brown"
                                                     next nil)))))
        (prinl ll)
        (list-update ll)
        (prinl ll))

Run it:

      $ txr linked-test.tl
      #S(lnode datum "how" next #S(lnode datum "now" next #S(lnode datum "brown" next nil)))
      lnode[0]->datum = how
      lnode[1]->datum = now
      lnode[2]->datum = brown
      #S(lnode datum "0:how" next #S(lnode datum "1:now" next #S(lnode datum "2:brown" next #S(lnode datum "3:cow!" next nil))))

We see that the list was altered with numeric prefixes on the strings, and a new node was added at the end.

Valgrind is completely clean:

      $ valgrind txr linked-test.tl
      ==6707== Memcheck, a memory error detector
      ==6707== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
      [ ... ]
      #S(lnode datum "how" next #S(lnode datum "now" next #S(lnode datum "brown" next nil)))
      lnode[0]->datum = how
      lnode[1]->datum = now
      lnode[2]->datum = brown
      #S(lnode datum "0:how" next #S(lnode datum "1:now" next #S(lnode datum "2:brown" next #S(lnode datum "3:cow!" next nil))))
      ==6707==
      ==6707== HEAP SUMMARY:
      ==6707==     in use at exit: 2,681,402 bytes in 9,346 blocks
      ==6707==   total heap usage: 14,827 allocs, 5,481 frees, 3,034,573 bytes allocated
      ==6707==
      ==6707== LEAK SUMMARY:
      ==6707==    definitely lost: 0 bytes in 0 blocks
      ==6707==    indirectly lost: 0 bytes in 0 blocks
      ==6707==      possibly lost: 1,286,596 bytes in 1,533 blocks
      ==6707==    still reachable: 1,394,806 bytes in 7,813 blocks
      ==6707==         suppressed: 0 bytes in 0 blocks
      ==6707== Rerun with --leak-check=full to see details of leaked memory
      ==6707==
      ==6707== For counts of detected and suppressed errors, rerun with: -v
      ==6707== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

Now with --free-all:

      0:sun-go:~/txr$ valgrind txr --free-all linked-test.tl
      ==6721== Memcheck, a memory error detector
      [ ... ]
      #S(lnode datum "how" next #S(lnode datum "now" next #S(lnode datum "brown" next nil)))
      lnode[0]->datum = how
      lnode[1]->datum = now
      lnode[2]->datum = brown
      #S(lnode datum "0:how" next #S(lnode datum "1:now" next #S(lnode datum "2:brown" next #S(lnode datum "3:cow!" next nil))))
      ==6721==
      ==6721== HEAP SUMMARY:
      ==6721==     in use at exit: 0 bytes in 0 blocks
      ==6721==   total heap usage: 14,830 allocs, 14,830 frees, 3,034,665 bytes allocated
      ==6721==
      ==6721== All heap blocks were freed -- no leaks are possible
      ==6721==
      ==6721== For counts of detected and suppressed errors, rerun with: -v
      ==6721== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

What's missing is circularity detection; we cannot pass a graph back and forth in this manner. If we pass a DAG structure, its shared structure will get expanded.

Since circularity detection is expensive, that would have to be something enabled by some sort of attribute mechanism.

</comment><comment><USER6> Interesting</comment>


Color-code terms in equations (2010) - <AUTHOR> http://labrigger.com/blog/2010/07/28/color-code-terms-in-equations/ <comment><USER2> Considering the time I see people spend wrestling with latex to get the formulas _juuuust riiight_ , people could start spelling things out in plain English--in half the time!

Having people outside of of that particular niche be able to understand the formula would just be gravy at that point.</comment>


India's 'Missed Call' Economy: An Untapped Goldmine - <AUTHOR> http://statspotting.com/2013/01/indias-missed-call-economy-an-untapped-goldmine/

<comment><USER2> Its worth noting that zipdial is using this missed call model very successfully. They were recently funded by 500 startup India and are planning international expansion.

</comment><comment><USER3> Sooner or later, there will be a charge for making the call, even if it wasn't answered. Making the call costs resources, and that needs to come from somewhere!

</comment><comment><USER4> Maybe that is the problem in India, everyone is trying to overide the system. If there is a loophole, let me exploit it.

<comment><AUTHOR> The positive spin is that we are still within the system, and maximizing the benefits at a low cost</comment></comment>


PHP vs. Node.js: The REAL statistics - <AUTHOR> http://www.prahladyeri.com/2014/06/php-vs-node-js-real-statistics/ <comment><USER2> > PHP might be still having the "king of web" status, but with node.js in the > town, I don't see that status staying for very long!

I doubt that. PHP will stay because of Path dependency, most webhosters provide PHP, because software is written in PHP. Software is written in PHP, because it is most widely supported. I like node better, though.

<comment><USER3> PHP is a lot easier to use. Which is pretty much why it's already king.

</comment></comment><comment><USER4> _The truth is out. node.js was faster than PHP by more 14 times! These results are astonishing. It simply means that node.js IS going to be THE de-facto standard for writing performance driven apps in the upcoming future, there is no doubt about it!_

While I enjoyed the benchmark test, I believe the conclusion that this is going to be de-facto as a result of it's high performance is a bit premature. There are so many factors (both right and wrong) that contribute to the widespread adoption to a language/framework/technology, performance often being less important than you'd at first think.

EDIT: I'm not suggestion node will or will not be a de-facto. I'm suggesting that the factors contributing to a languages widespread adoption are varied and complex.

<comment><AUTHOR> >>I'm suggesting that the factors contributing to a languages widespread adoption are varied and complex.

Yup, thats true, and being performance oriented isn't the only selling-point for node.js. The fact that all node apps are single-threaded (and thus conserve cpu capacity), combined with the functional-programming approach of javascript (which makes things like async I/O easier, don't hurt either.

<comment><USER5> I wouldn't say that single-threaded is a good thing, or that JavaScript is a functional language

<comment><AUTHOR> @sportanova - If callbacks are available for everything, then what is th need for multi-threading. You can handle everything in your own thread. When required, runtime will give you a callback.

JavaScript indeed has functional programming features. See: [https://en.wikipedia.org/wiki/Javascript](https://en.wikipedia.org/wiki/Javascript)

<comment><USER5> 1) While it's possible to take advantage of multiple cores with Node, it's easier to do it with a system that supports multi-threading out of the box. Node's event loop model (and interface, in the form of callbacks) works well for IO intensive purposes, but hardly obviates the use of multi-threading (especially when some CPU intensive processing comes into play). 2) I think functional programming languages are best viewed on a spectrum, and Javascript falls pretty low on the spectrum. Higher than Java, but far behind Scala and way, way behind Haskell. It has closures and higher order functions, but so does Java 8, and I wouldn't call Java a functional language. Off the top of my head it's missing immutability, referential transparency, pattern matching, lazy evaluation, tail recursion, and monadic structures. Just because a language has or is missing a few of these doesn't mean it is / isn't functional, but relatively speaking, Javascript is pretty low on the spectrum.

</comment></comment></comment></comment></comment><comment><USER6> It's not really a node vs php comparison, it's a node vs apache/php/unknown config comparison. HHVM can be 40-80 times faster than php for some benchmarks. Maybe try that?

<comment><USER3> nginx + FastCGI would probably put PHP on a more level playing field.

The test isn't completely invalid, but there is a lot more going on than what is described. It's a bit misleading to say that the server blocks waiting for file_get_contents. Actually, only the thread running the PHP script blocks. Other threads/processes can run. This experiment isn't quite a test of blocking versus non-blocking I/O. It's more a comparison of a heavy apache + PHP stack versus a lightweight, single-function node.js http server, which also probably benefits from non-blocking I/O. But non-blocking I/O is only one of many things that make the node script faster.

<comment><AUTHOR> >>Actually, only the thread running the PHP script blocks. Other threads/processes can run.

Yup. But imagine a situation that suddenly around 10000 requests have arrived at your server and sitting on the queue. Now that "one php thread" suddenly becomes a life-saver.

<comment><USER3> Indeed. That's what node.js is for. I was just nitpicking the article's explanation.

It would be interesting to see a breakdown of what makes the apache+PHP solution so slow. apache probably eats a lot of time by itself.

Are you using an opcode cache for PHP, like APC? The PHP script is being executed repeatedly, possibly being read, parsed and translated into bytecode each iteration. The Javascript gets compiled once into native code by v8 and just loops.

<comment><AUTHOR> The opcache is enabled and so is apc (though I don't think caching would matter much in this case).

As for apache eating time, nginx might run a bit faster than apache, but without a proper async I/O model built into the language itself (such as functional javascript), it would still be a band-aid solution.

<comment><USER3> True. I don't expect that nginx would make them equal. But it would be interesting to gather more evidence to support the hypothesis that async I/O is the difference.

</comment></comment></comment></comment></comment></comment><comment><USER7> Some time ago at school the teacher asked to make a crawler so, as always I do it in node.js and mongodb. The site to crawl has around 80K pages and 300 records on each page. My script crawled and saved all that data (80K * 300) in less than 1 hour with around 100 concurrent requests. The other students that wrote in ruby, python and php took days to do the same thing. Also they had to do by parts because if they requested all by once their pc would crash because of memory.

Also I didn't need any test because since the first time I ran node.js it was instant fast, not like php, ruby or python.

<comment><AUTHOR> Yup. And thats only because of precious CPU time saved by doing async I/O which the functional paradigm of JavaScript supports so well.

<comment><USER8> I really can't figure out whether you two guys are trolling. Thinking about it, I actually hope so, because everything else would be embarrassing.

</comment></comment></comment><comment><USER9> If you're really worried about performance just look at chicago boss or raw Yaws with Erlang and or Elixir.

The real question for most projects should be which is faster to develop with and has a better ecosystem.

</comment><comment><USER10> Wow, is this a parody? The lack of rockstar ninja references is confusing me.

"Hey guys, I was wondering if node really is faster than PHP, so I checked. It is! End of story."

</comment><comment><USER11> As long as there is software built in PHP which are used by huge numbers of people (Wordpress, Drupal, etc) then it will continue to be important to huge numbers of people.

For many of the people using these tools, the benchmarks don't matter. Most people do okay with their Wordpress or Drupal sites. Application developers understand that the limitations of attempting to use a CMS as a platform and they have wide choices (not just PHP or Node.)

</comment><comment><USER12> Just for fun, here's an unoptimized and un-async'd haskell version I threw together:

[http://lpaste.net/105557](http://lpaste.net/105557)

It takes about a second. I expected it to be faster than node and php, but it being this much faster makes me think I got something wrong. The files generated aren't all the same.

</comment><comment><USER13> The really approximated excluding concrete version information and using only read and write to disk commands statistics*

Personally i prefer [http://www.techempower.com/benchmarks/](http://www.techempower.com/benchmarks/) reply if anyone knows something better than that one

</comment><comment><USER14> I'd like to see the author make an attempt with the React library [1] in PHP as well. You don't have to abandon your language of choice to gain the ability to do asynchronous I/O.

[1] [http://reactphp.org/](http://reactphp.org/)</comment>


Comparison Analysis: Amazon CloudSearch vs Apache Solr  - <AUTHOR> http://harish11g.blogspot.in/2012/12/amazon-cloudsearch-vs-apache-solr.html

<comment><USER2> Kind of disappointed this doesn't include Solr's biggest competitor, elasticsearch[1].

[1] [http://elasticsearch.com](http://elasticsearch.com)</comment>


Microsoft's Windows Revenue Skids, Netbooks a Factor - <AUTHOR> http://seekingalpha.com/article/150946-microsoft-s-windows-revenue-skids-netbooks-a-factor <comment><USER2> Netbooks might be a factor in the other direction when it comes out. I've been an OSX user for years, but am planning on buying a netbook with Window7 when its available.

<comment><USER3> "but am planning on buying a netbook with Window7 when its available"

Me too, but I plan to rip it out and install a decent OS as soon as I turn it on ;-)

</comment></comment><comment><USER4> The days of $100+ OS are over. They'll milk it the most they can but they better embrace for impact.</comment>


Microsoft Allowed to Sue U.S. Government Over E-mail Surveillance - <AUTHOR> https://www.bloomberg.com/news/articles/2017-02-09/microsoft-can-pursue-suit-over-u-s-sneak-and-peek-searches <comment><USER2> Microsoft has an interesting stance here. As EU entities are major customers for both Windows and Azure, they're working quite hard to make sure to assure these parties that the US Government doesn't have undue influence over them.

They're adding secure links and solutions they cannot eavesdrop on to Azure, and getting pretty aggressive with pushing back against US surveillance.

The US benefits from EU privacy laws even without being there.

<comment><USER3> I agree with your analysis but what happens when China or a country like NK is the new economic powerhouse to please over EU ? Will US companies lobby the US the other way then ?

It's the Stallman argument again: don't defend privacy (free software) because it's convenient (hey, it's free and high tech enough), defend privacy because it's a principle that aligns with your ethics, not your economic interests.

<comment><USER4> I don't believe corporations can really have ethics in any meaningful way. For sure there are people who work there, often with a good amount of control, who may be ethical. On the whole, however, they report to boards or committees. In my opinion, it's much easier for a group of people to talk themselves out of acting ethically.

Having economic interests line up with corporate interests sounds as good as it gets, to me.

<comment><USER5> Companies often act ethical when the rules support this. If you tax pollution or other externalities then company's take them into account. And people generally prefer working with ethical companies which is a bonus. However when you don't tax pollution companies that act unethically have a huge advantage and win driving more unethical behavior.

</comment></comment></comment></comment><comment> awinter-py Their argument that cloud can't survive invasion of privacy is probably right, but ignoring government, cloud operators threaten the privacy of their own users.

G and MSFT both have stories from the early days when their employees went into cloud email accounts to check up on users. In G's case it was a rogue sysadmin stalking some high school kids, for MSFT it was looking at a journalist's hotmail acct to prosecute a leak.

The NSA has their own version of this, LOVEINT, where analysts stalk their significant others (or desireds or exes).

I can't think of any guarantee a company can provide to say this isn't happening, esp on social platforms. Crypto-based platforms _might_ do it (see the danish sugar beet auction, which uses a form of secure multiparty computation) but that's not MSFT's argument here. Also unclear how crypto platforms affect the ability to roll out new features and debug.

<comment><USER6> Back in summer 2006 between semesters, I worked at a Microsoft call center for Hotmail plus/premium (people/companies who paid premium account, and some ISP rebranded Hotmail accounts).

We had full access to billing, we also had full access to people's accounts. We could look at emails, look at deleted items weeks after del (restore in rare circumstances).

The thing is we didnt mess around. The tools to do this are heavily logged and audited.

There was an incident of guy looking into his gf account.

Within a week the guy was repramanded and let go.

<comment> awinter-py what did he do with his girlfriend's information, and how was it discovered that he looked?

Do you know if she was ever told? If yes, what was her reaction?

<comment><USER7> > > The tools to do this are heavily logged and audited.

> how was it discovered that he looked?

That's how they figured it out. Because someone who wasn't fulfilling a support ticket accessed an email account.

</comment></comment></comment></comment><comment><USER8> That the free speech arguments are going ahead is nice, but the judge also hammered another nail into the coffin of the 4th amendment:

 _Robart rejected the tech giant's argument that the so-called sneak-and-peek searches amount to an unlawful search and seizure of property. Former Attorney General Loretta Lynch had argued that federal law allows the Justice Department to obtain electronic communications without disclosure of a specific warrant if it would endanger an individual or an investigation._

<comment><USER9> It takes more than a judge's comments to change a constitution.

<comment><USER10> True, but a judge's job is to _interpret_ the Constitition.

<comment><USER9> Sure, but different judges are free to interpret the constitution without reference to specific interpretations. Therefore, it's not a nail in the coffin of the 4th amendment, it's just a temporary misuse (IMO).

<comment> k-mcgrady >> Sure, but different judges are free to interpret the constitution without reference to specific interpretations.

Aren't the lower courts subject to the interpretations of the higher courts?

<comment><USER9> For rulings, yes, though they can be overturned:

[http://money.howstuffworks.com/10-overturned-supreme- court-c...](http://money.howstuffworks.com/10-overturned-supreme-court- cases.htm)

Is what we're seeing here a ruling, or merely a dismissal of a claim before the trial starts?

</comment></comment></comment></comment></comment></comment><comment><USER11> Dog and pony show from enthusiastic PRISM partners.

"We've changed, honest!"

<comment><USER12> In the time of _fake news_ and _alternative facts_ , it is quite difficult to discern who is telling the truth and who isn't. After the revelations of programs like PRISM and XKeyscore it has become undeniable.

I think the general population is beginning to realize this. Obviously it takes time for people to understand things, but this information is years old at this point.

The question now is, "what do we do about it?"

<comment><USER13> > what do we do about it

Which _we_ are you asking about?

Since this is HN, I'm guessing you are asking about technologists and founders. I would say they should end surveillance on their users. Stop using ad-tech and analytics that tracks users. Log as little as you possibly can. Stop with the excessive rights grabs over your user's data.

But HN is also a lot of people like me - typical users. To them I would say use ad-blockers every where. Use HTTPS everywhere. When you are asked to give up privacy, consider what you are getting in return. For me, I'm comfortable with using Google services because it feels like I'm getting a lot. But with Windows 10, I'm giving up a lot of privacy for nothing in return.

I have no idea what to say about people like my parents. They have no problem with universal surveillance. Privacy is important only for protecting themselves from fraud. Same goes for my kids. They care even less about privacy than my parents.

<comment><USER14> re: windows 10... for most people it was a free major OS upgrade, increased security, modern browser with more frequent updates. I feel it was worth it as a free (as in direct monetary cost to me) upgrade.

I've got mixed feelings about the amount of telemetry data they collect.. I don't use cortana, and have disabled most of the options I can disable regarding this. That said, I mostly like the new UI/UX, and would prefer to have friends and family on a version of windows that is clearly better supported moving forward.

I wouldn't say that is _nothing_ ... I also run a mac laptop, and my most used home PC, is the htpc connected to my tv running Ubuntu. Android phone and tablet... I'm not only using windows, but can appreciate what it does offer in trade for that privacy concern.

<comment><USER13> I too like Windows 10. I just wish Microsoft would charge what they need to charge and make sharing optional. At first the enterprise edition had switches to turn off the data sharing but as I understand it, even enterprise edition users have to let Microsoft watch them work now.

You're right though - lots of people did get it as a free upgrade. I'm not sure I'd like Microsoft off the hook though since a significant number were tricked into it. Plus I think my point applies even more aptly to users that didn't upgrade to W10. Microsoft added the data sharing features into Windows 7 and 8. So they literally were essentially forced into sharing private data with Microsoft for nothing in return.

</comment></comment></comment></comment></comment><comment><USER15> It's very rare that I say this, but go Microsoft!

</comment><comment><USER16> We will have our Fourth Amendment. I'm actually not surprised it's Microsoft leading the charge here. After all, if they capitulate, their presence in the EU will be greatly diminished.

</comment><comment><USER17> It seems to me that requiring fixed end dates for the gag orders would greatly mitigate the problem. It would be possible to extend them, individually, for cause.

</comment><comment><USER18> I wonder how this will play out in other tech arenas.

</comment><comment><USER19> Some companies are smart and run EU & US operations completely separately. This is exactly why they're doing it.</comment>


Haskell Snap Framework templating 3000x faster with new release - <AUTHOR> http://snapframework.com/blog/2012/12/9/heist-0.10-released

<comment><USER2> 5 microsecs translates to 5000 nanos. Assuming an average CPI of 0.5 we have 10000 instructions to render the template. Or 30-50 main memory fetches. This isn't too shabby.

But the numbers before the improvement. Ouch. Those were bad.

<comment><USER3> I'm not sure measuring the number of _CPU instructions_ makes much sense anymore.

In my experience micro-optimizing code, the majority of time is spent waiting for memory (bandwidth waits or even worse: latency costs).

<comment><USER2> Precisely. This is why I mentioned the 30-50 mems. A typical main memory fetch is somewhere in the 100-200ns range on modern hardware. I tend to measure the effectiveness of an algorithm based on, roughly, how it accesses memory.

Caches play a role as well, of course. L1 is around 1ns sometimes even less. if you _hit_ L2 it is in the vicinity of 5-10ns. It easily translates to some 30-40 insns on modern hardware.

It also tells you that hunting for faster execution by compressing instructions down is not going to cut you that much extra speed nowadays. The key to fast programs is data representation. Good data representation.

</comment></comment></comment><comment><USER4> I wanted to ask how is that even possible, then read:

> However, we realized that a lot of the transformations could be done at load > time and preprocessed to an intermediate representation.

Lazy computation has its disadvantages. Still, impressive gain.

<comment><USER5> > Lazy computation has its disadvantages. Still, impressive gain.

After reading the article I don't see anything about lazy evaluation. What am I missing?

<comment><USER6> I don't think you ARE missing anything. From my reading, the gains are obtained by pre-computing the string concatenation a single time (rather than while rendering) for all cases in which it can be.

It's similar to moving something from runtime to compiletime (although those distinctions don't quite apply).

<comment><USER7> >I don't think you ARE missing anything. From my reading, the gains are obtained by pre-computing the string concatenation a single time (rather than while rendering) for all cases in which it can be.

Lazy evaluation: waiting until a computation _needs_ to be done to perform it.

Problem here: it was inefficient to do a particular computation at the very moment before it was needed.

So, how is this not a lazy evaluation problem?

<comment><USER8> Laziness is a specific property of how variable binding and application works in a language, which is not at issue here.

Unless I am mistaken, they didn't change the template language evaluation strategy from call-by-name to call-by-value.

They did change the implementation from an interpreter to a compiler, though.

<comment><USER4> I had the same line of thinking as SilasX. Conceptually, the change is that instead of deferring work to the last moment available, it is now done immediately. This is the very difference between lazy and eager computation. As I'm not a Haskeller, I didn't immediately realize the strong connotation with language features. Sorry for the confusion.

</comment></comment></comment></comment></comment></comment><comment><USER9> I find it interesting that Heist looks to be inspired by or based on XSLT, yet XSLT is not mentioned anywhere in the documentation. Is it just a happy coincidence?

<comment><USER10> My inspiration for Heist came from Lift's template system. That and FBML. Heist is essentially a generalized system for building domain specific markup languages.

</comment></comment><comment><AUTHOR> Independent of Haskell, Heist is one of my favorite HTML/XML template engines so it's great to see such huge advancements.

<comment><USER11> could you expand on why is it that you favor it?

I ask because I like the idea of a stateless xmlish template language, but I wonder what this offers over the zillion existing solutions.

"Separates view and business logic" and "enables DRY design" are valuable goals, but most template languages have them.

<comment><AUTHOR> Good question. A few reason:

1\. Heist allows you to define your own HTML/XML tags in the host language (Haskell in this case). This means you're only dealing with (an extended) XML document when doing layout and design so all the normal XML tools still work.

2\. Some popular template engines _try_ to separate logic and design but end up letting you cheat a little. Any time you want/have to cheat and put logic in the template, it really was a shortcoming of the template engine. In Heist, you can't cheat but you never want to.

3\. The reason #1 and #2 work is because Heist's "recursively applied splices" is just the right abstraction. My HTML templates end up looking just as pretty as well factored Haskell code. Heist makes the perfectionist in me happy.

In short, I would say you're right here: '"Separates view and business logic" and "enables DRY design" are valuable goals, but most template languages have them.' But just because most template languages have these goals doesn't mean they've achieved their goals. Heist, in my experience and opinion, does achieve these goals.

<comment><USER12> From their compiled heist docs [1]:

 _There are two things that compiled Heist loses: the ability to bind new splices on the fly at runtime and splice recursion/composability._

I haven't checked or read the doc thoroughly, but if it's what I think it means - all we get is hierarchal splices. Which is still a lot, but it's not quite as magical.

[1]: <http://snapframework.com/docs/tutorials/compiled-splices>

<comment><USER10> We still keep some of the magic by allowing you to run the old "interpreted" style splices at load time. These don't have access to dynamic data, but they do have recursion/composability. This combination retains most of the power while allowing a huge speed increase. It just means that to take advantage of both you have to structure things in a certain way.

At this point it seems to me that this structure also ends up being a desirable one for organizational reasons. But the jury is still out as far as whether there will still be reason to want more. We're aware that there might be good reasons to support this extra power and I have a pretty good idea of how it would be implemented. But I want to get more people using it in the real world before we address that issue.

</comment></comment></comment></comment></comment><comment><USER13> "Built for speed from the bottom up. Check out some benchmarks." from the frontpage of snapframework.com

"When we originally wrote Heist, speed was not our goal." from the link submitted here

Feel like a contradiction, couldn't they just say that it turned out fast enough on the first try?

<comment><USER14> I think that comment is referring to the framework / server, which is quite fast. Heist is a templating system authored by the same people that was not intended to be fast (though now it is). It is somewhat confusing, but I don't think the work they did on the framework/server (which is totally separate from Heist) should be discounted as "fast enough on the first try."

<comment><USER10> Correct. We always marketed Heist as a more experimental part of the framework as a whole. The server and associated API was initially our primary focus.

</comment></comment></comment><comment><USER15> so why did the api have to change? couldn't the compilation be done on first use? is the api change simply a change of the encapsulating monad (guessing wildly)?

not trying to bash haskell, but i think there's an interesting q about how well it (or any other language) can hide changing implementation details (particularly major ones like a compilation phase) behind an unchanging api.

or maybe that would have been possible, but the api changed for other reasons (the general cleanup)?

really interesting article btw. would have loved more detail... an explanation of introducing compilation in haskell with example would be pretty cool (pretty sure either pg or norvig has written one - with lisp - that i vaguely remember reading years ago).

<comment><USER10> The changes were significant because Heist isn't just an API. It's an inversion of control where you provide routines that get run for various parts of your DOM. They used to be functions that took a node and returned a list of nodes. In order to do the optimizations that we wanted to do we had to change the type signature of the callbacks that the users write to something that took a node and returned a special data structure.

We actually did preserve the old API, so you can actually migrate without making significant changes to your code. Most of those changes are because of the general cleanup. So maybe my statement about big breaking changes was misleading. They're big breaking changes IF you want the performance increase. Otherwise things still work the way they did before. In fact, the process of implementing this refactoring impressed upon me that the old paradigm was even more important that I initially realized.

If you're interested in more detail, check out the rest of the docs linked at the end. They describe the concepts in more detail with a focus on how to use them. In January I will also be giving a presentation to the New York Haskell Users Group (<http://www.meetup.com/NY-Haskell/>) about some of the things I learned while implementing this new approach and merging it back into the original Heist code base.

</comment></comment><comment><USER16> But this doesn't apply to splices that need data at runtime, like say pulled from a database right? Isn't that typically going to be 95% of your splices? The performance increase seems a bit overstated if it only applies to splices that are just simple substitutions.

<comment><USER10> This might apply to 95% of splices, but not 95% of your template. This particular benchmark does show the best case, but the typical case of a few dynamic splices will not affect things much because the page is still getting converted into a concatenative style and a ton of the splice processing of things like <bind>, <apply>, etc is happening at load time.</comment></comment>


Oracle Buys BlueKai - <AUTHOR> http://www.businessinsider.com/rumor-oracle-buying-bluekai-2014-2

<comment><USER2> $400M seems like a steal for Oracle. DMP's are finally becoming much more accessible to advertisers, agencies & publishers. We're only beginning to touch the surface for audience modeling capabilities and taking these learnings across multiple channels for direct targeting.</comment>