Ask HN: Why GraphQL APIs but no Datalog APIs? - <AUTHOR> Why doesn't anyone use Datalog (or another limited-by-design logic programming language) as the query language in a public Web API?  The adoption of GraphQL suggests demand for letting the API user write more sophisticated queries even at a performance cost per query.  Datalog would let the user do more, likely saving many HTTP round-trips.  A lot more research exists on optimizing Datalog than GraphQL.  It is unclear that the total impact on performance would be significantly negative compared to the typical GraphQL API.  If this line of thinking is valid, where are the Datalog API experiments? <comment><USER2> For context, I'm a frontend development lead who oversees a number of projects of different shape and size. My personal and very subjective opinion is it's simply hype. If you say "Datalog" to a frontend developer, they will either hear "obsolete" or just not know what you mean. If you say "GraphQL", they hear "+5 CV points". Obviously that is slightly tongue in cheek, but the marketing side of it is a very real factor.

Facebook currently holds one of the largest levers in frontend development — React. If they say the officially recommended way to do X with React is Y, you can be sure that Y will get at least moderate traction.

I'm sure they exist out in the wild, but personally I haven't seen a project where GraphQL really shines through. In my experience, HTTP2 and reasonably well designed RESTful endpoints are the right default to go with. There is the argument that building good APIs is hard, but I believe someone who has reasonable experience with a stable technology will outperform someone using a new tool for the job. If you're not familiar with it, you won't know what to look out for.

<comment><USER3> > I'm sure they exist out in the wild, but personally I haven't seen a project > where GraphQL really shines through. In my experience, HTTP2 and reasonably > well designed RESTful endpoints are the right default to go with. There is > the argument that building good APIs is hard, but I believe someone who has > reasonable experience with a stable technology will outperform someone using > a new tool for the job. If you're not familiar with it, you won't know what > to look out for.

GraphQL doesn't make a ton of sense to me, unless you're expecting many clients with significantly different query pattern requirements and prefer to take on performance and (maybe) security uncertainty & complexity to accommodate that more easily.

If you only expect one client (or a few very similar ones) and you want GraphQL-like division of labor just have your frontend and backend folks actually talk to each other and maybe have the backend folks also write a client library for their services (my preference, most of the time).

[EDIT] but you bet your ass I'm not putting up much of a fight against it if I'm in anything lower than a development lead or architect position—no use fighting that fight from poor footing when, as you mention, it's +5 CV points anyway, so who cares if it's maybe not the best solution :-)

<comment><USER4> > _" GraphQL doesn't make a ton of sense to me, unless you're expecting many > clients with significantly different query pattern requirements and prefer > to take on performance and (maybe) security uncertainty & complexity to > accommodate that more easily."_

My impression of it is that it's an "API" that's masquerading as a dumb relay, essentially giving almost datastore-level access to the frontend client (even though they don't market it that way). And this is done so that "Frontend devs" can just do "Frontend" things with access to all the data they might end up using. A bit snarky, but I do see it as a way to just "offload" the problem down the line, instead of thinking about the API-usage and what it means for potential frontend clients. That's the "old way" of doing things of course.

It fits in to the rest of the "movement" or trend currently happening: Thick (web) clients that do everything and the kitchen sink (including auth. See JWT in arb HTTP headers instead of cookies), SPAs so that there is no server side page rendering (this must all be done in the thick web client you see), document databases instead of SQL so the "frontend" devs don't have to worry about schemas, SPAs so that we break half the bloody internet because we now need to retrofit browser navigation using hashtags, everything has to be an "API" now because we can't convey data using server-rendering, etc.

Honestly, just that navigation thing alone is nuts and I can't believe we ever allowed it to be imposed on us. At this point, I'm _afraid_ to touch those buttons on a new website without first trying them to learn their specific behavior on that site.

<comment><USER5> You can handle proper URLs in a SPA if you know what you're doing. It's not something your average frontend dev would pay attention to though, sadly. Server-side "isomorphic" rendering of pages is also possible but a lot more complicated to resolve all the async requests that have to run to render the page.

<comment><USER6> Unless you use GraphQl of course!

</comment></comment></comment></comment></comment><comment> j-pb It's all about the data manipulation capabilities that our current programming languages and libraries provide. Datalog and GraphQL are both graph query languages, but they differ significantly in the datastructures they return.

GraphQL queries describe Tree unfoldings of graphs, and thus return trees.

Datalog describes recursive conjunctive queries on hypergraphs (relational model) without or limited negation, and thus return a set or bag of hypergraph edges.

The reason why GraphQL is so successful is that it fits well with Reacts data- model (trees) and the way it performs its efficient delta updates (tree walking). Furthermore its somewhat easier to implement (albeit not simpler). Consider that there are very few actual GraphQL query engines for actual data (e.g. DGraph DB), instead GraphQL backends implement resolver functions which compute the "graph"/tree on the fly, based on side effects. Resolvers are what you'd call a computable in prolog or datalog knowledge bases and even though I work full time on incremental query evaluation in said knowledge bases I don't have a clue on how to make those efficient without resorting to the Big Cannon of Differential Dataflow.

Datalog queries don't return trees, they return relations a.k.a. hypergraph edges, and that simply doesn't map well to the datastructures (maps/dicts and lists/vectors/arrays, JSON and other tree description formalisms) that we have in basically every mainstream programming language that isn't prolog or some variant on logic programming. So the query results are hard to work with, and you'd want a LINQ style embeddable datalog query engine in your language of choice to work with the returned data, which is a much bigger undertaking.

So to recap: Datalog is great if you have it everywhere and your language is build around it and hypergraphs, but alas most languages we use today are build around trees, and graphql is a tree language.

</comment><comment><USER7> That's a very justified question!

In my opinion, Datalog or — more generally — Prolog is where many querying APIs will eventually arrive, because this syntax has many nice advantages: It is very convenient, expressive and readable, programs and queries can be easily parsed and analyzed with Prolog's built-in mechanisms, there is an ISO standard for it etc.

When semantic web formalisms were discussed, Prolog was sometimes mentioned. However, it is so far not very widely used as a querying or modeling language even for semantic web applications. Recent advances in Prolog implementations may increase the uptake of this syntax.

Also, public perception of Datalog and Prolog lags somewhat behind recent developments in these areas and what these technologies can actually do or what they even are. For example, here on HN, many posted articles that claim to be about Datalog often show snippets and examples that use completely different notations which are no longer a subset of ISO Prolog and therefore also not Datalog. Thus, they also do not immediately benefit from recent advances in Prolog systems in that they cannot directly be parsed and interpreted by them. It may take some time to advertise these improvements, and increase interest in these formalisms.

<comment><USER8> What's the best example of modern Datalog?

<comment><USER7> A good way to try Datalog is to use a state-of-the-art Prolog system that supports _SLG resolution_ (tabling).

SLG resolution is an evaluation strategy of Prolog programs that provides favourable termination properties compared to SLDNF resolution which is Prolog's default execution strategy.

A great example for this is XSB Prolog, since it has pioneered and refined many of these techniques and serves as an example and benchmark for many other Prolog systems in this area:

[https://xsb.com/xsb-prolog](https://xsb.com/xsb-prolog)

The nice thing is that this approach allows you to use Datalog, and at the same time gives you a Turing-complete programming language that is comparatively easy to grasp once you understand Datalog, while sharing many of the same advantages and being syntactically only a slight extension of it.

<comment><USER9> A turning complete query language sounds like a recipe for DoS attacks.

<comment><USER10> It is only Turing complete if you allow new rules. Usually a client would only be allowed to formulate query goals.

In that sense, GraphQL is Turing complete as well, as the underlying implementation could just simulate a Turing machine.

<comment><USER11> What I read from your post here is that GraphQL is safe by default and would require effort to make it dangerous while Datalog is dangerous by default and would require configuration to make it safe. I know which one I'd rather put on a server.

<comment><USER10> You have read that incorrectly. A Datalog/Prolog query does not define new rules. That's like calling arbitrary code on remote machines. You would never allow that.

But IF you were allowed to define new rules, THEN you could simulate a turing machine or create infinite reductions.

</comment></comment></comment></comment></comment></comment></comment><comment><USER12> Prolog (Datalog) has always been the gold standard for query languages. GraphQL is merely the latest attempt to reinvent it by people who obviously don't know what came before (or don't care). This might be because CS programs are no longer teaching Prolog, but I think it's more likely because one doesn't get career advancement points for using something "old" unless one reinvents it and renames it so it seems "new."

<comment><USER13> I _know_ Prolog (from college) and SPARQL (from hobbyist dinking around with SemWeb). I love DSLs and weird languages. I should be almost the ideal Datalog user, so how did it stay completely off my radar until this post?

<comment><USER14> Because it's a niche tangent of an already niche language.

I'm only aware of it because a databases course taught it (relational algebra to datalog to SQL) and I thought it was superb, and a couple of times since I've tried to find if I can use it, but it seems to suffer from the combination of age + academia + proprietariness that makes such things hard to discover and use. (Compared to graphql and plethora of open source libs on Github, etc.)

</comment></comment></comment><comment><USER15> You are asking for a rational explanation for why something better is less popular than something that is worse. The answer almost certainly is "just because", and more specifically "because Facebook made GraphQL, not Datalog".

Graphql had a lot of hype around it when it was first announced. It so happens that I had just learned Prolog at the time and was personally bemoaning why SQL was so popular when a Prolog-like solution would be so much better. A month or so later comes all the podcasts and articles on HN from Facebook devs about GraphQL.

Datalog is a really great technology, and the idea that you can guarantee that each statement terminates is very powerful.

GraphQL seems to be a great technology when you have an extremely specific requirements. I personally would certainly use it if i had those requirements, but IMO it seems totally unjustified for your average tech company.

</comment><comment><USER16> Hyperfiddle (my startup) is a datalog API experiment, wrapped up in a low-code tool. We vertically integrate Datomic, a datalog database for AWS. Can you imagine GraphQL ever being low-code?

[http://www.hyperfiddle.net/](http://www.hyperfiddle.net/)

[https://gist.github.com/dustingetz/654e502340070280ab9744723...](https://gist.github.com/dustingetz/654e502340070280ab9744723a8ae250)

<comment><USER17> That is really cool. I'll have to check this out this week.

FYI Your HTTPS cert is tied to cloudfront and not configured correctly.

</comment></comment><comment><USER18> Ultimately GraphQL is popular because the query ergonomics fit well with the component architecture everyone is using these days, and once you're familiar with it you can be extremely productive.

<comment><USER19> sorry not a frontend dev, where can i get basic info about 'component architecture'

<comment><USER20> Modern SPA frameworks are usually built on components. You can combine components to define a tree of elements that represents your UI. This pattern is also often referred to as a Declarative UI pattern.

GraphQL works well with declarative UIs, because you can define fragments and combine them to create queries that best match the data the current set of components you are loading require. In this way, it's flexible in a way REST isn't (at least typically) and syncs well with Declarative UIs.

I can't speak to datalog though.

</comment></comment></comment><comment><USER21> This is actually fairly common in Clojure annd Clojure script. Sometimes using the Datomic DB as a backend for it.

This is an older presentation but its all about doing Datalog from the client to the DB.

See: [https://www.youtube.com/watch?v=aI0zVzzoK_E](https://www.youtube.com/watch?v=aI0zVzzoK_E)

But if you go threw the Clojure world, there is Datalog all over the place, both client and server. Check out Datascript:

[https://github.com/tonsky/datascript](https://github.com/tonsky/datascript)

This full-stack framework: [http://book.fulcrologic.com/](http://book.fulcrologic.com/)

<comment><USER22> > This is actually fairly common in Clojure annd Clojure script. Sometimes > using the Datomic DB as a backend for it. What do you mean by "this" which > is fairly common? Datascript is a Datalog implementation, however you can't > just bring your query from backend to frontend, not only for incompatiblity > but also for security issues. And Datascript is not that popular among > Clojurescript apps.

Fulcro is defintely the go to framework. It makes use of EQL, which is a superior alternate to Graphql, for server-client communication.

</comment></comment><comment><USER23> It's not clear to me that GraphQL and Datalog solve the same problems. GraphQL, to me, is fundamentally about designing the interface between a client and server, decoupled from the logic that satisfies requests. I think it's actually not a query language at all, any more than HTTP is a query language. Datalog is very much a query engine. And it's not super clear to me that the services exposed by a server map cleanly to a logic model. GraphQL requires no such purity.

<comment><USER24> That: GraphQL is a mechanism for clients to specify the data document they want, and for servers to specify the documents they can serve. Datalog, or SQL for that matter, is a mechanism to expose data within a database. The GraphQL data model must be geared towards the end client point of view, e.g. some internal IDs can be hidden. Datalog on the other hand exposes the internal model of a data store, which is domain driven not client driven. Also you do not want end client devices to have the full power of datalog or SQL on your database, or else be ready to always watch those crazy queries that bring your server down to a crawl. GraphQL at least provides an indirection point where you can apply some form of resource control.

</comment></comment><comment><USER25> I will say something that is commonly known. a lot of frontend software dev revolves around hype. if React is the in thing and fb dev's are promoting Graphql, then everyone will go towards that. I don't know much about Datalog, but JSonAPiSpec meets most of things that people tout Graphql for. + if you're a person trying to keep employed then you've to follow the trends .

<comment><USER26> React came out in 2013. That's six years and counting of "hype".

</comment></comment><comment><USER27> We currently looking for a powerful query language for a distributed event sourcing system. It would have to be embeddable in rust.

Is there a site where I can see more complex datalog queries and how they compare to other query languages? All the examples I have found so far were relatively trivial and basically just assumed that you knew that datalog was super powerful.

A comparison of how to solve problem X, Y, Z in datalog, graphql and SQL would go a long way to convince people that it is a superior alternative.

</comment><comment><USER28> As a company that runs both Django Rest Framework and GraphQL backends, there is one phenomenal argument in favor of GQL: Apollo Client. When you just want a simple, normalized reactive data store it's really amazing (even though the Apollo kids seem to break it with every minor release). Loads of redux boilerplate go right out the window.

<comment><USER29> Per the last comment, please check out our new official Redux Toolkit package. It includes utilities to simplify several common Redux use cases, including store setup, defining reducers, immutable update logic, and even creating entire "slices" of state at once without writing any action creators or action types by hand:

[https://redux-toolkit.js.org](https://redux-toolkit.js.org)

Not saying it's a replacement for Apollo (since Redux itself doesn't provide anything around data fetching), but we specifically created RTK to eliminate the concerns around "Redux boilerplate".

<comment><USER30> Still requires boilerplate for thunks, which in my case is most redux actions. Also I think you still don't support for specifying same reducer for many actions (for example as an array - [https://github.com/piotrwitek/typesafe- actions](https://github.com/piotrwitek/typesafe-actions)). The immutable update logic however is amazing and I am glad I got introduced to it by you!

<comment><USER29> Not sure what you mean by "boilerplate for thunks". You write thunks the same way you always have.

And yes, `createSlice` explicitly supports responding to actions that were defined elsewhere in the app, using the `extraReducers` argument:

[https://redux-toolkit.js.org/api/createSlice#extrareducers](https://redux- toolkit.js.org/api/createSlice#extrareducers)

Although... are you saying that you want one reducer function to handle many different actions in one slice? Just define the reducer outside the slice itself, and pass it as the handler for each action:

        createSlice({
            name: "someSlice",
            initialState,
            reducers: {
                action1: someCaseReducer,
                action2: someCaseReducer,
            }
        })

Did something similar in my own app recently at work, where multiple slices all needed to handle loading data similarly. So, I wrote a couple generic reducers, and passed them in to each of the slices.

</comment></comment></comment></comment><comment><USER31> TerminusDB does use a Datalog-like language as the query language for its public web API. Datalog has long seemed to me to be the obvious "next step" for query languages as it enables richer manipulations than those provided by vanilla SQL-like languages in a clean fashion.

However, as j-pb noted in another comment:

"GraphQL queries describe Tree unfoldings of graphs, and thus return trees.

Datalog describes recursive conjunctive queries on hypergraphs (relational model) without or limited negation, and thus return a set or bag of hypergraph edges."

TerminusDB allows you to use both approaches. Due to the way we have designed our graphs, with a strongly typed schema langage, you can extract sub-graphs as unfolded trees using a special predicate exposed in our datalog language.

This approach (hopefully) gives the best of both worlds, allowing graph traversal and extraction of objects/documents in a single framework, all communicating information via JSON-LD.

<comment><USER32> Sounds reasonable to me :-)

</comment></comment><comment><USER33> Well, there's EQL and various implementations that use datalog under the hood.

[https://edn-query-language.org/eql/1.0.0/what-is-eql.html](https://edn-query- language.org/eql/1.0.0/what-is-eql.html)

<comment><USER22> EQL is surely a great alternative to Graphql because of namespaced keywords and rich data structure (in contrast of Graphql's string base). It is also inspired by Datomic's Pull API. However, it's a specification and the most popular imlementation (Pathom) has nothing to do with Datalog.

<comment><USER34> The original question made this mistake probably because the poster didn't know it but Pathom is the actual alternative to GraphQL-based APIs. Datalog solves a subtly different problem of querying the data, while Pathom/GraphQL/REST is about exposing the data (interfacing?) as API.

It's more like:

Pathom vs GraphQL-based API (vs REST)

and

Datalog vs SQL (vs NoSQL)

The mainstream is in the process of migrating from REST to GraphQL, but I think Pathom could be the black horse in this race. Just watch the last part of Wilker Silva's The Maximal Graph talk to see the potential: [https://youtu.be/IS3i3DTUnAI?t=2080](https://youtu.be/IS3i3DTUnAI?t=2080)

</comment></comment></comment><comment><USER35> > It is unclear that the total impact on performance would be significantly > negative compared to the typical GraphQL API. Some of the problems also > apply to GraphQL, people just need to constrain the query complexity in the > end.

But yeah, it would bite Datalog more if not designed well enough, and it's harder to tackle Datalog than to GraphQL.

Other thoughts on this topic:

I love GraphQL but I would love Datalog more if the toolchain is as good as GraphQL.

There's a spectrum on API of simplicity and power -- on the left, there is HTTP/REST which could be seen as a primitive/naive way of describing API it's not consumer-driven at all. On the right there's Datalog which is a powerful way of describing API, it can maximize the flexibility of both the server and the client.

Things like 'map' and 'filter' became well-known to average programmers not very long time ago, though the elite programmers already knew them well half a centry ago. Datalog is definitely a very good idea for API, but I'm afraid it wound't be as popular as GraphQL if Facebook chose this route instead.

That being said, the industry always adopts things in a very passive manner, usually because the current solution is too painful to use, or almost couldn't solve the problem on new problem scale. There's a documentary[0] for GraphQL, the creators had enough of REST, they ended up with inventing GraphQL. It originally was to solving their data fetching problems, in the meantime it would solve others too. Meanwhile, it pushed the common understanding of API for average programmer from left to right on the spectrum. If it's close enough to the right, more and more people would discover Datalog and buzz other people to use it.

Just like type system, it's hard to expect the industry have a deep understanding on the topic itself. Type systems similar to Java dominates the industry for a long time, though ML already existed almost half a centry. Recently, languages like TypeScript tookoff because people need to type existing JavaScript. Then people would discover that the type system they used to is far from enough to describe existing code design. It's like a pandora box, new type operator would came out every release, until people realize there's a spectrum of type system (Lambda Cube[1]), and Dependent T ype would be on the same spot as Datalog on the spectrum.

[0] [https://youtu.be/783ccP__No8?t=645](https://youtu.be/783ccP__No8?t=645) [1] [https://en.wikipedia.org/wiki/Lambda_cube](https://en.wikipedia.org/wiki/Lambda_cube)

</comment><comment><USER36> How do I implement Datalog in my application? Crickets...

<comment><USER37> Depending on the stack, you can use some open source libraries. Datascript is quite robust, and useable from javascript/clojurescript. [https://github.com/tonsky/datascript](https://github.com/tonsky/datascript)

<comment><USER38> "An immutable in-memory database and Datalog query engine"

Most uses of GraphQL is to expose existing data from a relational database through an API, so although this looks like an interesting project, is doesn't look like it would be a contender against any GraphQL implementation.

I'd argue there are two faulty assumptions in the OP: 1. "GraphQL serves the same use case as Datalog.", and 2. "Datalog is better at that use case."

GraphQL is not a general purpose query language! There are practically no query operations at all built into the specification - it's all about field selection and parameter passing. At heart it's more like a RPC protocol geared towards requesting tree-like data. You would evaluate GraphQL against options like plain REST, OpenAPI, gRPC, not against Datatalog.

A more relevant question to ask is why Datalog never took over for _SQL_ , but it would be harder to pretend that SQLs dominance is due to it being the current passing fad.

</comment></comment></comment><comment><USER39> I've got this idea that you can chuck JMESPath into the range header on a standard json endpoint for ghetto graphql

</comment><comment><USER40> For me the most important thing about graphql is composability.

Child component can communicate their data requirements to the parent and parent can include them in the query and feed the data to the children as it arrives.

Don't know if Datalog has anything for that.</comment>


Ask HN: Why do I see LSD related news often in HN? - <AUTHOR> <comment><USER2> LSD has been consumed in hacker circles for decades, especially Silicon valley - 60s and 70s were both time when LSD was widespread and software world started heating up. And since hackers are intellectually curious, LSD is quite an attractive drug. Pretty simple.

<comment><USER3> For anyone interested in exploring this history in depth, I can warmly recommend Fred Turner's book _From Counterculture to Cyberculture: Stewart Brand, the Whole Earth Network, and the Rise of Digital Utopianism_. Very interesting stuff.

<comment><USER4> The Whole Earth stuff is fascinating. If you can get hold of the Catalogues they're full of interesting details.

People may want to read The Cuckoo's Egg (Cliff Stoll) and The Eudaemonic Pie (sometimes called The Newtonion Casino in other countries) to get the flavour if this bit of history.

</comment></comment></comment><comment><USER5> It's not really there. You're hallucinating.

</comment><comment><USER6> see: Silicon Valley microdosing LSD</comment>


Millions Had Risen Out of Poverty. Coronavirus Is Pulling Them Back - <AUTHOR> https://www.nytimes.com/2020/04/30/world/asia/coronavirus-poverty-unemployment.html <comment><USER2> Opening the economy will kill people. Keeping it closed will also kill people. It's complicated. Is anyone modeling this?

Back of the envelope: if 500 million people lose two years of life [1], that is 1 billion years lost. Assuming Coronavirus takes 10 years of life on average from 10 million globally, that is 100 million years lost.

[1] According to this article, poverty reduces lifespans in the range of 10-15 years. [https://www.thelancet.com/journals/langlo/article/PIIS2214-1...](https://www.thelancet.com/journals/langlo/article/PIIS2214-109X\(18\)30327-9/fulltext)</comment>


India : Rejecting a job offer? Get ready to pay penalty - <AUTHOR> http://economictimes.indiatimes.com/news/news-by-industry/jobs/rejecting-a-job-offer-get-ready-to-pay-penalty/

<comment><USER2> Title is misleading - penalty applies to reneging on a signed contract, not rejecting the offer.

Link is broken. Instead: [http://articles.economictimes.indiatimes.com/2013-08-23/news...](http://articles.economictimes.indiatimes.com/2013-08-23/news/41440580_1_job- offer-penalty-clause-offer-letter)</comment>


The Cutting Room Floor: Unearthing Unused Content from Video Games - <AUTHOR> https://tcrf.net/The_Cutting_Room_Floor <comment><USER2> This is pretty cool! Randomly came across this tidbit, regarding one of my favorite Apple II games: "While Karateka only takes up Side A, Side B contains a copy of the game modified to play upside-down. According to Jordan Mechner, this was a practical joke by Brøderbund so tech support could have the delight of telling someone that their game was upside-down because...they put the disc in upside-down. Duh."

<comment><USER3> Were you aware that Mechner released an ebook that was basically a compilation of his journals while he developed Karateka? Considering it's one of your favorite Apple II games, you might find it interesting. I did just as a game developer. I never even heard of the game, I think I ran into it because he did the same with Prince of Persia.

It's called "The Making of Karateka". You can find it on Amazon. (Also "The Making of Prince of Persia")

<comment><USER2> Thanks, I'll check that out!

</comment></comment></comment><comment><USER4> All of my favorite games ever are the 2D-era RPGs that I didn't play when I was young: Fallout 2, Chrono Trigger, Zelda Link to the Past. While CT and Zelda are relatively polished, Fallout 2 stands out as a piece of programming crap: buggy and crashes every so often, contents were cut left and right -- but boy, do I love that game. I feel like 30% of the contents were cut in the final version.

Thankfully that game has an incredible amount of people who love it, and it must be known to most people who play Fallout 2 by now: the "recommended" way to play Fallout 2 was to install the Restoration Patch. It restores all the cut contents back, and makes the game feel way more completed. Fallout 2 RP is one of the best examples of community patches I have seen.

<comment><USER5> Re: Fallout 2, I never played it, started at Fallout 3, but one thing Ive observed playing Fallout 3, New Vegas, 4, Oblivion and Skyrim, Bethesdas quality sucks. They may be good at creating a good rpg world and story, but the stability and quality of the underlying software sucks. From the crashes (which made New Vegas nearly unplayable at launch) to the stupidity of the AI followers (I need your help-wait where are you? Did I accidentally kill you? Nope, followed randomly shows up half our later like nothing was wrong).

<comment><USER6> I agree with your point regardless but just a side note that New Vegas was actually dev'd by Obsidian and not Bethesda Game Studios like the rest of your examples.

<comment><USER7> But they used a lot of tech (like the game engine) from Bethesda Studio to develop New Vegas. (a bit like they developped KOTOR 2 and NWN 2 using tech from Bioware)

</comment></comment></comment></comment><comment><USER8> This site has an impressive collection of in-game anti-piracy measures that is worth checking out [1].

From Donkey Kong Country 2 (SNES):

 _Checksum Verification - At the beginning of a level, a checksum is run on 544 bytes worth of code and data which includes all of the above tests, the PAL /NTSC region check and various initialization routines. If the checksum generated doesn't match the hardcoded value it compares against (i.e., the anti-piracy routines were tampered with), RAM address $7E0AFD, which holds the total number of screens in the level, is decremented by one. This punishes the player by causing the screen to stop scrolling just before reaching the goal in a level._[2]

It's so brilliant, yet so evil.

[1] [https://tcrf.net/Category:Games_with_anti- piracy_methods](https://tcrf.net/Category:Games_with_anti-piracy_methods)

[2] [https://tcrf.net/Donkey_Kong_Country_2:_Diddy%27s_Kong_Quest...](https://tcrf.net/Donkey_Kong_Country_2:_Diddy%27s_Kong_Quest_\(SNES\))

<comment><USER9> Hm, interesting. Long time ago, I intentionally tried cracking games to detect anti-piracy methods, and Donkey Kong Countrty 2 was one of those, and I triggered the empty SRAM check while doing so (and added that to the TCRF page). I checked the game with more SRAM than the cartridge normally has, and... nothing??? Checking that is a very typical routine in SNES games, and yet... I simply assumed due to that it's the only check, and an incompetent one at that -- first, it would be more evil to let the game continue with no save RAM only for saves to not work, second, why not check for more RAM.

But then it turns out there are actually were more anti-piracy methods, and they are quite evil. Wow. I'm not surprised I did not find those considering how those worked however, I had no reason to change the ROM to add a crack intro.

</comment></comment><comment><USER10> In case anyone is interested in learning more on the site's backstory, I did a profile on the site and its main operator a couple of years ago:

[https://tedium.co/2016/06/16/hidden-content-video-games- cutt...](https://tedium.co/2016/06/16/hidden-content-video-games-cutting-room- floor/)

<comment><USER11> This is exactly the sort offbeat content I love from your blog. Thanks for writing it!

</comment></comment><comment><USER12> This is a great site I've explored for a while now. Some of the entries are shockingly detailed. Majora's Mask, for N64, for instance has a very long and organized page ([https://tcrf.net/The_Legend_of_Zelda:_Majora%27s_Mask](https://tcrf.net/The_Legend_of_Zelda:_Majora%27s_Mask)) as well as separate pages for the debug and prototype versions ([https://tcrf.net/Proto:The_Legend_of_Zelda:_Majora%27s_Mask](https://tcrf.net/Proto:The_Legend_of_Zelda:_Majora%27s_Mask)). The site Unseen 64 ([https://www.unseen64.net/](https://www.unseen64.net/)) has a similar collection of info on betas/cancelled games.

</comment><comment><USER13> Yeah, I've been a member there for a few years now. Great site, lots of interesting information about unused content in games. I personally worked on the pages for Wario Land 3, Wario Land 4 and Zelda Breath of the Wild, and will contribute a few more things later too.

</comment><comment><USER14> This is really cool, thanks for sharing OP! I've spent the last hour reading up on the featured article, on Pokemon Gold and Silver

</comment><comment><USER15> Does anyone have recommendations on where to learn how to discover this unused content?</comment>


Air France pilots faced a cascade of failures - <AUTHOR> http://www.latimes.com/news/nationworld/world/la-fg-brazil-air-crash13-2009jun13,0,3778145.story <comment><USER2> Very poor article:

"As that system failed, the plane's computers automatically reduced the pilots' ability to move the plane's control surfaces -- technically known as going to "alternate flight law."

Ditchey said alternate flight law is meant to protect the plane by reducing the ability of pilots to make a mistake, but ultimately it also may limit the ability of pilots to save the plane."

Um, no. The alternate law has reduced protections against the pilot screwing things up and would allow the pilot to, for example, stall the aircraft. It's Normal Law that limits what the pilots can get up to.

If the journalist has thought about it for one second it would have been obvious. No one down in Blagnac is going to follow the thinking "Hmm. In the case that we've lost computer control of the aircraft let's introduce a new set of laws that limit what the pilot can do"

</comment><comment><USER3> Link to single page:

[http://www.latimes.com/news/nationworld/world/la-fg- brazil-a...](http://www.latimes.com/news/nationworld/world/la-fg-brazil-air- crash13-2009jun13,0,4880794,full.story)</comment>


Robocom: a game for programmers - <AUTHOR> http://robocom.rrobek.de/

<comment><USER2> If you're into this kind of thing, you should check out Battlecode (<http://www.battlecode.org>). It's primarily an MIT competition, but in recent years we've opened it up to the general public. Also, several YC alums are former devs.

</comment><comment><USER3> > Your goal is to put the other robots out of action. But not with weapons!

Hey! What's the fun in that? :P

</comment><comment><USER4> It was a disappointment to check the downloads page and discover that there isn't anything other than a windows version available.

EDIT: It does seem to run on wine.

</comment><comment><USER5> Hi, I'm the guy who invented RoboCom back in 97. Yes, RoboCom Workshop is Windows only (unfortunately), but there is a compatible interpreter named RoboTour which has been ported to various OS. It's not at fancy as RoboCom Workshop but it basically does the same.

The most exciting news about RoboCom is that I have started to re-write RoboCom as a web application in pure JavaScript (actually CoffeeScript). It runs in any modern web browser. The compiler and the simulator core both somewhat work, but it's very early alpha. The thing is that I have currently no time to continue the development because I founded a company which keeps me super-busy... but expect it to happen some day :)

</comment><comment><USER6> It's some kind of Core Wars clone, right?

<comment><USER7> There are similarities, but it's a very different game. Robocom takes place on a 2D grid, and more resembles a fight between microorganisms. Each bot has its own individual memory banks, but these can be overwritten by other bots. So you might inject an instruction into an enemy to tell it to die, but a more effective strategy might be to instruct the enemy bot to infect others for several turns and _then_ die.

</comment></comment><comment><USER8> News hasn't been updated there since 2009..

</comment><comment><USER9> With all the 0x10C stuff that has been produced lately, I remembered I saw a game where you have to program AI to battle your opponents, but I couldn't remember the name.

Now I may have something to do :)

<comment><USER10> Maybe it's robocode? <http://robocode.sourceforge.net/>

<comment><USER9> Not quite, it ran on DOS and was coded in an assembly-like language. Chances are it actually was robocom.

</comment></comment></comment><comment><USER11> Oh wow, I used to tinker with this back in high school. Unfortunately, I could never convince my friends to join in, and I eventually lost interest. I'm impressed that it's still around.

</comment><comment><USER12> Linux version would be nice.

<comment><AUTHOR> It runs perfectly under wine.

</comment></comment><comment><USER13> Karel.turnRight();

<comment><USER14> That was honestly my first thought when I read about this. I could see Mehran playing this in his spare time. It would be a pretty great end-of-semester project too, having a competition like this in the class. Everyone would know Java, but they would have to learn a new language by building off what they already know.</comment></comment>


XP Sales end June 30: Time to dump Windows?  - <AUTHOR> http://news.yahoo.com/s/infoworld/20080217/tc_infoworld/95335

<comment><USER2> this is obviously a rhetorical question.</comment>


Tech titans should value military service - <AUTHOR> http://www.usatoday.com/story/tech/2015/10/25/voices-tech-titans-should-value-military-service/71635480/ <comment><USER2> Leave it to USA Today to produce drivel like this. From the article, "But that misses an important point. It's the intangible qualities of military experience that are the most valuable. If you survive the Army's Ranger School or a year of combat in Anbar province, Iraq, you have probably demonstrated a level of intestinal fortitude that goes beyond what is required to roll out the latest iPhone app."

Aside from the fact that the above is laughably specious, what exactly are those intangibles? One important intangible is that the military places a tremendous emphasis on rule-orientation and institutional rigidity. I can hardly think of less desirable qualities in most employees, especially in a startup.</comment>


Ask HN: Is 'search' a solved problem? - <AUTHOR> I remember the time (not so long ago) when 'search' seemed to be the hottest topic in the industry. We had the rise of Google and competitors. There were search startups. Open source projects like Lucene and Solr were in the news. There were books published, blogs, conferences..
And now it seems that the industry have moved on. There is a million papers/books/blogs/online courses/video lectures/meetups about ML/AI, but I can't seem to find anything current on search.
What are the good resources to learn the fundamentals of search and keep up with the current happenings in that space? Not SEO, but more from the computer science/engineering point of view? <comment><USER3> "Search" is too broad to ever be solved. That's like "solving entropy".

Google focused on a specific subset — you enter a few keywords or a phrase, and the machine returns the top ~10 links to pre-existing (indexed) web pages. But that's not all there is to search!

Challenges:

1\. Intranets: internal documents, typically in different modalities (FAQs, support cases, wikis, public pages) and across diverse storages that evolved throughout the years via acquisitions and osmosis.

2\. Clustering: you don't have any keywords, but rather want to find how a particular document (legal template, its clause section) evolved over time. You want to _avoid_ using keywords. Search for similar documents or document sections. Find similarity between two documents that is based on semantics rather than query keywords. Applications: eDiscovery, contract management…

3\. SME & Intent: "relevant result" means different things in different domains, or even different aspects of a single domain. Google is doing an amazing job with their "single search box", but there are industries (for example, HR) where search _precision_ matters much more than _recall_. More elaborate, focused, domain-specific facets or even dialogue systems make sense there.

Commercial plug: we built a search solution focused around semantic search (in the "machine learning and vectors" sense, not "sematic web and RDFs" sense), [https://scaletext.ai](https://scaletext.ai). It's still early days in that our clients are all over the place, but to say Google/Lucene solved search is patently false.

<comment><USER4> Definitely valid point (that there are very different types of "search", not just one universal way to do things), but:

2) Google does not do simple keyword matching, and certainly has a strong sense of "semantics".

3) Try searching "Cheap hotels in San Francisco" or "Plumber jobs in Chicago" in Google. Just because it's a single search box does not mean all results are generated/displayed in the same way.

<comment><USER3> Absolutely. Determining query intent is an ancient, well researched and still active domain. And Google publishes their results regularly (kudos to them).

My point was more that encoding the relevant signals into a single general- purpose search box (a sort-of natural language) is an inherently noisy, ambiguous process. When you know what kind of search you want, it's better to factor out the relevant parameters and feedback loops explicitly, give them a clear UI and search flow. Rather than have users fumble with double quotes and double-guessing the query parser.

<comment><USER5> @Radim - is this something you are doing with your current startup? p.s. kudos from another fan of gensim.

</comment></comment></comment></comment><comment><USER6> If anything, it's an abandoned problem. A lot of companies bought really expensive enterprise search systems, which are sitting dormant because the results are so bad.

With advances in spamming, internet/email search is getting to be a harder problem every year.

I remember when Google was quite effective in finding what I need, but nowadays it's dismal. As an example, I googled for "storename return policy", and got page after page of results. All of them were tagged "missing storename", so just randomly picked return policies from other stores.

Search is their bread and butter, and they're probably keenly aware of the diminishing quality of results. I'd love to hear what's caused the recent trend for results that are missing a few of the most crucial keywords. Probably over-enthusiastically trying to filter out keyword farms.

<comment><USER7> Lately, it has felt like Google has become more optimized for less intelligent/tech-savvy users (i.e. people who search using natural language e.g. "tell me the store policy for target please"). In these cases many of the words used are either irrelevant or counterproductive.

I've noticed Facebook's search has started catering to the lowest common denominator as well. Searching for (made-up example) "Louis Potter" used to give all exact matches priority. Current results look something like 1) Louis Potter 2) Luis Porter 3) Louis Potters 4) Louis Potter (#2). I don't like when search engines assume that I'm misspelling words/names, and it would be nice if they adapted for individual behavior in this regard.

<comment><USER8> facebook used to let you do direct knowledge graph searches

"restaurants liked by people who like Joe's Pancakes and Green Dragon Mexican Pizza"

"movies liked by people who are friends with people who like Back to the Future and Arrested Development"

"people who live nearby and like Green Dragon Mexican Pizza and Arrested Development"

"friends of friends of friends who like Arrested Development and Joe's Pancakes"

"Friends of Megan Albert friends who are women named "Erin" Chipotle employ"

I think you can see why they removed it.

this URL might lead to something like "restaurants liked by people who like Snow White" [https://www.facebook.com/search/130104022710/likers/pages- li...](https://www.facebook.com/search/130104022710/likers/pages- liked/273819889375819/places/intersect/)

<comment><USER8> [https://code.facebook.com/posts/153625638171563/under-the- ho...](https://code.facebook.com/posts/153625638171563/under-the-hood- indexing-and-ranking-in-graph-search/)

</comment></comment></comment></comment><comment><USER9> Given that google is the de facto standard ATM and often returns irrelevant results, I'd wager that search _isn 't_ solved.

"That does not by itself mean there's room for a new search engine, but lately when using Google search I've found myself nostalgic for the old days, when Google was true to its own slightly aspy self. Google used to give me a page of the right answers, fast, with no clutter. Now the results seem inspired by the Scientologist principle that what's true is what's true for you. And the pages don't have the clean, sparse feel they used to. Google search results used to look like the output of a Unix utility. Now if I accidentally put the cursor in the wrong place, anything might happen." [http://www.paulgraham.com/ambitious.html](http://www.paulgraham.com/ambitious.html)

<comment><USER10> I would suppose that this change is less google changing and more the web changing as people pump out content marketing and SEO. Also might be "olden days were greener", I sure remember a lot of keyword stuffing back in the day.

<comment><USER11> You're probably right about that. Although I can point to one very specific way the olden days were better: a Google search used to turn up lots of forum posts, which are often the most useful results on a given topic. These days I almost never see forum posts of any kind turned up a search results (except stack overflow, thankfully). I'm not sure what caused the change but I find useful information is much more likely to be buried by low-information wikihow-type results and corporate landing pages.

<comment><USER12> Possibly because most forums seem to have died? Or maybe they died because they got excluded from search results?

</comment></comment></comment></comment><comment><USER13> Not really, no. Some parts of search (Internet) have been cornered/solved by big players (e.g. Google), but many other parts (e.g. search in Intranets) are still an open problem. No one has found a solution that's as simple as PageRank for Intranets. No one has found a solution for the "the author of half of these documents is the intern who wrote the template"-problem and many other things. There are good products out there, but a "Google for Non- Internet" is still far away.

p.s.: All the bad searches on the various products I use or the websites of some companies make me almost want to get back into search. Because even for things that ARE solved in the technical sense there seems to be no "out of the box" solution or people would use it.

edit: To expand a bit - all the search solutions I've seen which weren't for Internet search were more or less bespoke, so you needed a project to get something decent. Sure, you can install a plain Lucene/Solr, but Lucene/Solr cannot understand the way your data works, which parts are important or if you want to show results which are older further down (or not!). They have decent defaults for the "common case", but you have to tune them for every customer/installation for good results and that makes it non-scalable. And being scalable without effort is usually one of the requirements people have for something to be "solved".

<comment><USER14> So true, I can ask Google a range of verbal questions and get great answers, mostly what-is-the-fact questions.

At work we are nowhere near asking an intranet "show me the last pentesting report for Tony's new website" or "show me the change requests for the failed change this quarter". I know exactly what I want, and how to ask, bit I will not get the results I want.

</comment></comment><comment><USER15> We are still in the stone age when it comes to search.

Ask Google who played the mens semifinals of Wimbeldon three years ago and Google will tell you it indexed 6 million pages to provide a link that may or may not have the 4 names I am looking for. Why is it doing all this pointless work? And why is it that dumb in 2018?

We have got so used to what it does that lot of people have stopped asking questions about how it does things and wether all the stuff it does is required.

Wolframalpha, Freebase/SemanticWeb/Wikidata/dbpedia approaches, NLP/NLU are still very underdeveloped and untapped.

Having open and distributed indexes like we see in nature with DNA is also totally unexplored because of Google type centralised index monopolies in various domains. It just takes a Gig or so to store a local offline index off all Wikipedia or Stackoverflow pages. And given the massive RAM and hard disks everyone has these days why aren't we seeing sophisticated local offline search apps?

The internet is getting exponentially more noisy day by day and in many ways its easier to find quality info going through a top notch library's index than wading through Google's. So there are lots of blindspots and areas to explore in search right now imho.

<comment><USER16> I think these sort of queries are solved.. if you know the categories of sites that index information and are able to scroll and process text, images and information quickly.

My first query string idea was 'men semifinals Wimbeldon 2015 wiki' and the resulting page contains the list in a nice format.

This is because I have the context that wiki pages would contain this sort of information. Google and others are getting better at processing more vague queries (like 'three years ago'), but I do agree we are nowhere close to being able to ask general questions. Knowing how to use the tools like google search (and other searches) and really advanced queries syntax is a force multiplier/enabler.

<comment><USER17> > if you know the categories of sites that index information and are able to > scroll and process text, images and information quickly.

That's a job for computers to do.

</comment></comment></comment><comment><USER18> If you just want a primer on how to think about adding search to a product, this piece by Max Grigorev is a great starting point: [https://medium.com/startup-grind/what-every-software- enginee...](https://medium.com/startup-grind/what-every-software-engineer- should-know-about-search-27d1df99f80d) It's ostensibly for the engineer, but actually feels more like it's written from the POV of a product manager.

</comment><comment><USER19> It's not that people have moved on. It's that the entire culture of the ecosystem is built upon a narrative that Google is an all-powerful machine that cannot be stopped or contested. So people don't try to compete, and if they do, they will be ridiculed for it.

And for what good reason? Certainly not because of past attempts. In fact, Google has bought some startups that were involved with search.

I'd be interested in knowing what you find. There are bound to be many relevant texts not labeled as search, if you know what you are looking for. Perhaps I would start with web crawlers and go from there.

<comment><USER20> As a side topic, is there a useful web search engine that uses a fundamentally different approach to Google, e.g. aren't using backlinks as a ranking signal?

When Google's approach isn't giving me an answer, it'd be nice to try a search that wasn't based on a discoverability feedback loop.

</comment></comment><comment><USER21> My personal view is that there should be dedicated search providers for specific areas, such as academic, or technical, or news.

That way each provider can focus on building a good platform with clever machine learning tailored to _that_ dataset.

We also need the return of proper Boolean operators and complex nested queries. Yes, Joe Public will never use them, but a _lot_ of people who search the internet as part of their jobs, or just have deep interests and would love to have all those advanced search features back, and be able over to override the 'fuzzy logic' that generic Search engines such as Google enforce on us.

I also disagree that all sites need to be mobile first. If I have a site that provides software for the Enterprise Sector, Google will still penalise me if my site isn't responsive, even though my target market is IT professional sitting in front of powerful laptops/desktops.

As many would agree, Google have too much power in the Search space, and they have basically dictated to the world how Search should be done, whether they are actually right or not.

</comment><comment><USER22> No, it's not solved. Google utterly fails at it.

I'm a person with simple needs. I mostly search in English, sometimes in Russian (which is my native language) and sometimes in Japanese (which I'm learning). These languages use completely disjoint sets of characters so it should be obvious to Google what I'm trying to do (since I'm logged in and they know all about me). There's also a setting that allows me to pick languages that Google should prefer when searching (English is always selected). Now there's a little problem:

1) If I choose any language other than English, Google prefers websites in that language over English _even if my query is in English_. If I'm searching for programming-related stuff, I don't need it poorly translated into my native language, or any other language! I want the original, on the first page of search results. So any language other than English gets switched off.

2) If Japanese is not switched on, Google thinks that any query that consists only from kanji (a subset of Japanese characters that are also used in Chinese) is in Chinese language, so I get pages and pages of Chinese websites before I see any Japanese ones. Since I can't read Chinese at all, it's completely useless. Now, you might say: of course Google has no way to know if I wanted Japanese, not Chinese. Oh, it knows. I can search for a name of a Japanese person, and Google will display a sidebar with their English Wikipedia article, date of birth and nationality and everything, and yet all the search results will still be in Chinese.

<comment><USER23> > utterly fails

> $74b revenue

> 80% market share

<comment><USER24> Social networking's industry is also a great success, look at Facebook.

</comment></comment></comment><comment><USER25> I can give a list of things I'd like in a search engine:

1) Sticky topics - if I'm at work and I type Flow I am way more likely to mean Facebook's js library than I am Flow energy.

2) Different views on the information (Grid view/masonary that makes sense)

3) Ability to search within a set of search results

4) Ability to customise the algorithm with other programs I can write/plugins

5) Many more things for programmers and advanced "omni bar" style features that allow me to type shortcuts and autocomplete things.

6) More automation of programmer stuff - if I type a number and a unix timestamp it should show me a date etc. for possibly millions of things. Same for unicode char of a hammer, url decode, etc. etc.

7) Clean integration with your OS search that makes sense.

I'm aware Duck Duck Go does some of these things (relatively badly). I think I'll give it another go then and see if it's better.

<comment><USER26> Great list, +1 to everything, +10 to #1, #3.

</comment></comment><comment><USER27> Far from it. I frequently have issues even with search on Google or Amazon, who invested millions into that. Dealing with people being imprecise in naming and spelling, different contexts and personalized results is still barely touched or non-existent, and definitely not solved.

Lucene provides building blocks, not a solution, and not even all of them.

Search on a small scale may seem working if you have phrases that are mostly unique, like recent movie titles or celebrity names. Go beyond that and its a mess.

<comment><USER28> The 'solution' you're looking for also isn't necessarily the 'solution' Google and Amazon are looking for. You're looking for an answer to a question, or to find a thing, they're looking to maximize ad revenue and sales. So the answer to your search is tempered by what gets them the most money. _That_ is what they've spent millions trying to figure out.

</comment></comment><comment><USER29> I don't see 'search,' aka information retrieval (IR), as a solved problem. I went to the last SIG IR conference in Tokyo, and yes, the heavy hitters in the field were there to promote and present their latest research. It is no doubt a very active research field using machine learning techniques. Reading the papers published could give a view of the (academic) state of the art.

Whether it is a good business strategy to challenge Google head-on is another question. Whether there is a sure way to learn and engineer said systems on extremely large scales is another question.

</comment><comment><USER30> In 1994 there were people who believed search was 'solved' thanks to Yahoo! providing a comprehensive index of the Web (which actually was doable for a couple of years) Then the Internet exploded in size so in 1996 there were people who believed search was 'solved' thanks to AltaVista. The Internet continued to grow and thanks to early SEO techniques like keyword stuffing, an opening existed for Google to fill when they 'solved' search with 'The Algorithm.' Now we're in the midst of search being 'solved' again via ML/DNN's. And I'm sure it will be 'solved' again and again in years to come. So to answer your question about if it's solved, you'd first have to specify which time? :-) Search is a very, very large space and will likely not be solved anytime soon.

Also, don't confuse the business/tech press definition of solved (i.e. a dominant player raking in large piles of money and shutting out competition) with said problem actually being solved as in there does not exist a better way to attack the problem. Granted, when the business/tech world considers a problem solved, this often just means that the lions share of (known) financial incentives are no longer 'low-hanging fruit'... until a challenger figures out a way to blow up the incumbents business model.

</comment><comment><USER31> DARPA/IARPA now invests in search 2.0: Instead of asking it to retrieve stored information, you can instruct an agent/search bot to perform tasks for you.

For instance, one should be able to search for: "who is the leader of this IRC hacker group?" "where can heroine be bought on the deep web?" "Is there women trafficking going on behind that log-in wall?" and then an intelligent agent is dispatched, avoiding/crossing roadblocks, like log-in forms, and will eventually bring you the answer.

Coursera has more on the basics, like: [https://www.coursera.org/learn/text- retrieval](https://www.coursera.org/learn/text-retrieval)

OpenAI has more on the current happenings of creating more intelligent search bots: [https://github.com/openai/universe](https://github.com/openai/universe)

Other possible future research areas in information retrieval include being able to search for services ("Where is cheapest taxi service for current location?") and an integration with IOT.

</comment><comment><USER32> I tend to think the search is often _the last resort_ and indicative of the other navigation system being broken. When people can reach the info they need in a more organized way quickly, they'd probably do so. Therefore full text search has to cover every residual task; it's bound to be messy.

</comment><comment><USER33> Elastic and Solr (Lucene) with the help of various other dbs or graph dbs etc get you farther along but you really have to combine machine learning to get things in context which requires more data in certain domains so it requires someone like me that goes things alone to dive into many disciplines. It is not straight forward and results are not yes or no, more like 70% good and the rest is subjective.

The most challenging parts are if you have many dimensions at the same time such as location / full text / user preferences / social filters / permissions etc. These are the problems that make my life suck right now as you cannot simply not have joins for some things or not have graph relations etc etc. Case in point following feeds with many dimensions so you need a pipeline approach in stages.

</comment><comment><USER34> I've learned that search is very domain-specific and it's only "solved" if you can pin down what "relevant" means for your corpus of documents.

In terms of fundamentals, I'd suggest reading about tf-idf, which is the basis of Lucene (which powers Solr and Elasticsearch).

</comment><comment><USER35> Google's good for 1 - 3 key words / phrases but anything longer can still be difficult.

Regardless of query length it's strange that I often find the second result better than the first one.

Finding something in Gmail can be a real challenge, I wonder if would be possible to unleash Algolia on your inbox.

</comment><comment><USER36> I've confronted my PhD supervisor (Professor of Library and Information Science) with this statement once, and she almost went berserk. Her take is that free text search is approaching the solved problem stage, but almost all other search isn't.

</comment><comment><USER37> You might find this interesting [https://dynamicguy.com/post/160792434677/1-search-engine- not...](https://dynamicguy.com/post/160792434677/1-search-engine-not-good- anymore)

</comment><comment><USER38> The search problem is connected to the spam filtering problem which is an ever advancing arms race - it's never solved and depends on the new schemes spammer come up with. So search itself is never a solved problem.

</comment><comment><USER39> I wish there were a search engine that could figure out the groups the search results belong to. Sometimes when searching for swift programming language, a certain singer also makes an appearance. Or when searching for physics related things, I get that Olivia Newton John song. Like I was a search engine that displays some sort of Venn diagram (not quite but it's close) that let's me hone in on my results.

To be honest, I think that I would actually want an engine that scrapes fewer sites but good sites and tries to understand them better. Also regular expressions.

</comment><comment><USER40> Great topic. Part of the issue is probably a transition from algorithmic approaches to data-driven approaches. What did previous users search for and click on? Existing companies have a huge advantage from years of data, and not the kind of advantage that others can learn from (compare to publishing a better algorithm). Another factor may be that parts of the problem can separated out and are studied on their own, such as natural language processing.

<comment><USER41> >What did previous users search for and click on?

This is why google often is bad for searching tech things. You get often very old useless links for a topic

</comment></comment><comment><USER42> One aspect that I don't see mentioned is that APIs for search have basically disappeared from the major players, likely due to them being expensive (programs can hammer an API faster than a human with a text box) and lacking a revenue model (no human may be involved, making advertising useless).

This has caused "search" to devolve into "human using Web browser types natural language in box, human is presented with results (some semantic, most just links to things other people wrote, some advertisments around the side), human reads through results to see if any are useful to them".

This is certainly useful, and I rely it all the time, but it's not the pinnacle of what search could be. It's like if `grep` didn't pipe to stdout, but instead popped up an alert box for each line like "Your query '.*' matched the line 'foo'. Try the new McDonalds saver menu today! [Next/Cancel]", it would still be useful, but nowhere near as useful as piping to stdout.

Many years ago Google provided an API to their search engines, which applications could build on to be more "smart". This could have paved the way to much better software: for example, imagine a prolog system where all of Google's knowledge could be used by the calculations.

That path was mostly abandoned since there's no scalable incentive to make it operate, much like the semantic Web. Rather than opening up databases to empower others, it's much more profitable to keep them walled off behind a few limited, pay-per-use interfaces (e.g. "paying" by showing a human some adverts). Attempts to bypass or abstract over these interfaces are hit with rate limiters, Recaptcha checks, etc.

</comment><comment><USER43> Lucene/Solr, Elastic Search and Algolia did a great job creating search tools and services and this extinguished the thirst of the masses. I don't think it's a solved problem, it's just a problem that has commercially viable solutions. When it comes to resources, I've found valuable knowledge in Lucene/Solr forums and mailing lists, back in the day. It's worth a read.

</comment><comment><USER44> I think search is not only not solved, but a failed concept overall.

It's too hard to find relevant information from the gazillions of pages based only on a few words.

I think search needs to be replaced by some kind of indexing/ontology/knowledge organization system, and then maybe only be applied in the "last mile" of a person's 'search' for relevant information

</comment><comment><USER45> I'm increasingly thinking that a problem with search is inconsistent and nonstandard (or nonexistent) conventions, protocols, and APIs.

It seems to me that a fair bit of the search problem could be addressed by sites themselves serving wordlists, tuples, statistically improbable terms (there's another term for this that's escaping me), etc., rather than consenting to being heavily crawled by numerous spiders.

Vastly improved content metadata (particularly for largely fixed "article" content), including author, date, and (a reliable) topical categorisation would help. For realtime information, APIs are probably more reasonable than text search, though those would likely be front-ended by specific applications.

This still leaves the very nontrivial problems of reputation, relevance, black-hat SEO, and information manipulation (propaganda, misinformation, misdirection, disinformation), and just plain street-grade idiocy.

But several elements of this strike me as amenable to either localised or distributed solutions.

</comment><comment><USER46> We're trying. Kindly see [https://millionshort.com](https://millionshort.com)

<comment><USER47> Gives good results. Thanks for providing the link.

</comment></comment><comment><USER48> I'd like a search for things that I've read or seen in the last few days. Or years.

[https://twitter.com/patrickc/status/953011978217205760](https://twitter.com/patrickc/status/953011978217205760)

<comment><USER41> [https://yacy.net/en/](https://yacy.net/en/)

Yacy can do it. You can use it as a proxy and it does index all visited pages. (I tryed Yacy, but never this feature).

</comment></comment><comment><USER49> It depends what you are looking for. I have the hardest time searching for laptops that meet all of the specs I want, for instance.

Takes days.

Minimum screen brightness. 1440p touch. Needs USB-A. 8th gen quad core. 20+ watt TDP. etc. Not too thin, not too thick. decent graphics.

<comment><USER50> In the EU you can use [https://geizhals.eu/?cat=nb](https://geizhals.eu/?cat=nb) which is pretty decent. I'm using that regularly to check options.

Is something like this not available in the US/other territories? If not: here is your opportunity :-D

</comment></comment><comment><USER51> Self-plug: I recently launched a dedicated news search engine at [https://yetigogo.com](https://yetigogo.com) — Based on my personal needs, but it works pretty well for tracking any current event.

<comment><USER52> I hope you are ready for the "link tax"...

</comment></comment><comment><USER53> Searching is easy enough but scoring the value of what it finds is not! Is an answer on Stackoverflow valuable because the poster has 100K reputation? (no!) Was the information valid 10 years ago but completely irrelevant now? Is this programming blog post very specific to Drupal and not relevant to other PHP frameworks? Was it information taken from somewhere else? (I would rather send traffic to the original).

Another problem is how to search for something when you don't understand it enough to search for it or you can't think of distinct enough words or phrases to search for it.

</comment><comment><USER54> Internet search has been solved for about 10 years.

Evidence: Before it sold off to Bing, Yahoo search was quantifiably better than google for a few years (in blind tests where you rip off the branding).

No one cared, because google was good enough.

Having said that, I use duck duck go these days, and occasionally spot check using google. I think the google results have become unusable because they too aggressively map to related concepts, and otherwise second guess what I've typed, but there have been endless debates about that on HN, and it's essentially decided by the user's taste.

<comment><USER55> Couldn't you use quotation marks to search for phrases? I do that all the time. The search is not verbatim, Google certainly stems the content of quotation mark phrases, but I was under the impression that related concepts are not included.

I'm not claiming to be sure about that, it's a subjective impression, so if someone can explain better, I'd be very interested. Since I use Google for work (more often than Google scholar), I'd hate not to find what I'm looking for, but generally Google results seem to be much better than others. It's the main reason why I don't use DuckDuckGo, ixQuick, Yandex, Bing.

<comment><USER54> Aggressively ""'ing can help, but I think that also turns off useful stuff, like word stemming, when what I want it to do is not map to some other more popular term.

I'm sure people have been trained to use google effectively, but I switched years ago, and find it harder than the alternatives.

To be clear, this is all nitpicking. The last time this came up on HN the only query anyone found that actually showed practical differences between the two was the acronym FOSS.

Google rewrote to "free open source software", with no quotes, giving poor quality results (eg bsd software).

Ddg gave the definition of foss, and relatively few results pointing at actual software, because that acronym was (is?) obscure.

</comment></comment></comment><comment><USER56> It's solved, but as many has cited not to a satisfactory degree. Modern search engines have extremely short query deadlines (users won't wait more than a couple of seconds when searching), which gives low precision results.

This book is very informative: [https://nlp.stanford.edu/IR- book/](https://nlp.stanford.edu/IR-book/)

edit: I should say it's slightly outdated because of lack of "big data" and how search companies currently deals with huge amount of data.

</comment><comment><USER57> I discovered this recently: [https://typesense.org/](https://typesense.org/) Very nicely done and might be useful for learning from.

</comment><comment><USER58> The field of Information Retrieval has largely moved to multi-modal retrieval (search across video, audio, text), linked data, question-answering and so on.

But document retrieval (classic search as you describe it) is not a hot topic anymore. That does not mean there are not people working on constantly improving document retrieval: Google Scholar returns 17,800 results for "document retrieval" in 2018 and 34 results with it in title. So it is in widespread use but not the focus of the field, I would say.

</comment><comment><USER59> Absolutely not. Frequently I try searching for things from years ago by describing them into a search engine, and SEO spam floods my results. I usually end up asking a human.

</comment><comment><USER60> I would love to see a search engine that only returns websites that have no JavaScript present (or some other artificial way of excluding major, "modern" websites).

</comment><comment><USER61> In a sense, it deteriorated for some basic requirements. Remember the the time, when you were able to use boolean operators and _exact_ sequences on Google reliably.

</comment><comment><USER62> If you squint a bit, you will find every computation is a kind of search. You are searching for the "output" based on the given "input". Ex: Deep learning is just searching for the weights. What people usually called search is where the search space is explicitly defined (a set of records in db etc) but in most cases the search space is there just that it is implicitly defined in the computation problem description.

</comment><comment><USER63> I don't think it is a solve problem. It just seems daunting or difficult to take on Billion dollar companies in the space.

</comment><comment><USER64> The default search engine in people's web browsers is a solved problem. Search itself, not so much.

</comment><comment><USER65> Search is very much not "solved" as other have pointed out. I'll add that when google started search was considered pretty much "solved", and Yahoo turned down the option of buying google as they considered their search good enough.

</comment><comment><USER66> To break into this space, I would recommend starting with a subset that people want to search. Like the facebook model of only being available for some colleges.

Some ideas:

\- Only academic papers

\- Only news sources

\- Only hacker topics

\- Only financial topics

\- Only small bloggers

\- Only literal keyword search which Google discontinued

Get traction in that domain, then build out from there.

</comment><comment><USER67> I see opportunity for niche search engines, there are several areas that google does not do well in on purpose it seems.

I think the truly hard part is that so many people accept whatever default is already there. If they get an android phone - they use the search box there. If they are using chrome browser, whatever input box is there on the first screen is obviously the url bar and use that (you and I may know the difference, the average user doesn't care, it's one less click to just type 'google' into the url box in the center of the page, of fbook or whatever, then google brings up the url you were going to (not searching).

This is why I think there is much less hype about competing in this space. Unless there is a thing forcing companies to put other browsers and search boxes on phones, tablets and chromebooks like the microsoft IE debacle so long ago.. then trying to be the next google is impossible, even if you had better results, better tech, etc.

Regardless of that, I think it's quite possible to make much better niche search engines and get them used. If ten micro engines could make 1% of googles revenues each, that would be a decent amount of money in my neck of the woods.

I'd like to see other people post more sources about search tech in general, several searches last year only brought a few info bits on what it may cost to create an index of the net - someone posted some numbers using servers bought off ebay and a rack at hurricane I think - had some numbers for the cost of servers to pull a new index every month or so?

Certainly the tech and costs have changed since that was published, but not much I've seen.

I'm pretty excited at this project posted recently: [https://news.ycombinator.com/item?id=16976941](https://news.ycombinator.com/item?id=16976941) ( Show HN: A search engine that doesn't track you, where users vote for results (github.com) )

I am hoping to get some people together to make a less persnickety and fussbudgety search option for people who don't want to be babysit with censoring kids gloves when looking for fun things.

If anyone wants to make a couple adults only engines, or ones that are more fun, let me know.

Average people talk in slang and cut up about less high brow things, the big G gives rank to the college papers and deranks for so many things, it's on the road to being the next yellow pages and sciences journal, but not the place to go when you want fun things anymore.

</comment><comment><USER68> There are big opportunities precisely because the field seems dead.

(1) The first big story is the dominance of Google. With an advertising- centered model, Google has a reason to degrade result quality. If you get trained to scroll down to find the real results and you found them good, you might avoid touching any of the ads (hard to do because they cover so much of the screen.)

(2) The web is 95% Javascript and 95% Spam -- getting useful results at all requires fairly strict 'censorship' and vast resources if you want to compete on Google's ground. No serious competitor will come in with a different model, nothing will change unless you have a search engine that _YOU_ pay for and not the advertisers.

(3) "Desktop search" is discredited in most peoples minds. Your OS might have added it as a feature back in 1995, but you've kept it turned off because it slows down your computer and never finds what you are looking for. Result quality is an issue, but the #1 perception here is that the indexing process harms the user experience. In the era of multicore, NVMe, etc. can this be changed?

(4) "Website search" is also discredited. Product search commonly works, but search on most web sites is so bad that people are trained to just search on Google. Thus you have very few chances to change people's minds.

(5) There is a big literature (the TREC conference) but there is something profoundly depressing about it. It was one of the first big competitions, but unlike the SAT Solver competition or Imagenet it was not associated with a rapid improvement of technology but rather a painful slog through the mud. If you start reading it at the beginning or in the middle somewhere you will find that 20 or so things that you thought were sure bets to improve relevance don't work. If you read the cliff note's to the first 10 years written by the organizer, you find out that there was an interesting discovery made 5 years in...

(6) The BM25 ranking function which has two tunable parameters. BM25 was a huge advance because it can be tuned to comparable rank documents that are highly variable in size. BM25 is built into Elastic Search, but nobody will give you any advise how to tune those parameters...

(7) Because they don't follow the relevance evaluation protocol in TREC; this is badly flawed, but the data exists, and going from naive tfidf to tuned up BM25 or information theoretic approach (also implemented in Elasticsearch) will put up better numbers AND seem more relevant to end users.

(8) An open-source project to do that evaluation on Lucene got started but never made a project; I have talked with Enterprise Search vendors who were very aware of points 5-7 but did not tune up their search because it was easier to sell customers on having hundreds of "connectors".

(9) The mainstream of TREC (it has broken into many flavors) and IR research has been getting high recall at low precision. Maybe that's because when Gerard Salton was messing around with punched cards at Cornell, 70 abstracts was a lot of documents. Patent searchers and paralegals are interested in deep recall, other people aren't.

(10) A major flaw in the mainstream TREC approach is that they are trying to tune up the wrong function: the ideal relevance score is a probability estimator of how likely the document is to be relevant.

(11) Google and Bing have made noises about personalized search but they don't really do it. They are both stuck at 70% relevance for the first result because of their limits in inferring user intent. The real relevance function has the user's context as an input variable, but sampling by that thins the data points to where it can't be approached as a "big data" problem. "Personalization" works for advertisers who don't know your real intent but are willing to pay for a 5% chance you may click, but not for you where you will feel misunderstood (primed to get irrationally angry) 95% of the time.

<comment><USER69> Re: desktop search. What. I see "lay users" in meetings with projectors/TVs using the search function in the windows menu to find documents all the time! My org is named something like XQWK, so the files they want to show have the letters XQWK. It's pretty natural to them.

</comment></comment><comment><USER70> 'Search' might be solved, but 'find' isn't.

</comment><comment><USER71> Why doesn't browsers have a search for bookmarked pages ? It's because they get payed by Google to have their users use Google search instead.

</comment><comment><USER72> I remember when I used to hear all the time about the semantic web. It sounded more and more like mind reading.

Computers can not read minds.

</comment><comment><USER73> I remember reading about this exact topic somewhere but I can't seem to find the link.

<comment><USER19> Here are my comments on the subject from December which might be of interest. [https://www.quora.com/Is-it-possible-to-beat- Google](https://www.quora.com/Is-it-possible-to-beat-Google)

</comment></comment><comment><USER74> The solution we need is search that doesn't rely on an information monopoly like Google.

</comment><comment><USER75> (Disclaimer: I am an Apache Solr committer and popularizer)

Search is interesting! And it is important to differentiate the web search (Google) and domain-specific search (Solr, Elasticsearch, recent release of [http://vespa.ai/](http://vespa.ai/)). You cannot tune Google to your domain needs and understanding.

For domain-specific search, the basics are there. Even the fancy "basics". It is now very easy to add search to one's stack. In fact, Solr is in so many stacks, it is not even mentioned much anymore. But we still get the contributions back from Cloudera, Bloomberg, Alfresco, etc.

So, the cutting edge in Search is now on personalization, relevancy-tuning, indexing non-text content (music, images, etc), multi-word semantic search, graph traversal and, yes, Machine-Learning. See, for example, [https://lucene.apache.org/solr/guide/7_3/learning-to- rank.ht...](https://lucene.apache.org/solr/guide/7_3/learning-to-rank.html)

In fact, the Solr conference that used to be called Lucene/Solr Revolution is now Activate and has focus on ML/AI because the topics are really starting to overlap ([https://activate-conf.com/](https://activate-conf.com/)). You can see the interesting topics from last conference: [https://www.youtube.com/playlist?list=PLU6n9Voqu_1FMt0C-tVNF...](https://www.youtube.com/playlist?list=PLU6n9Voqu_1FMt0C-tVNFK0PBqWhTb2Nv)

Learning (Solr at least) is a different issue. There are so many features now that the Reference Guide is absolutely enormous. And the demo schemas are still a bit of a kitchen sync, making it look more complicated than it needs to be. And, the last comprehensive book was several versions back. Again, that's because Solr is big and is growing really fast still...

Actually that's why I chose to be a popularizer within the Solr community and focus on making it easier for beginners to start.

See, for example, my latest presentation slides at: [https://www.slideshare.net/arafalov/rapid-solr-schema- develo...](https://www.slideshare.net/arafalov/rapid-solr-schema-development- phone-directory) and the backing configuration repo: [https://github.com/arafalov/solr- presentation-2018-may](https://github.com/arafalov/solr-presentation-2018-may) (includes smallest viable useful schema)

(tl;dr) Search is still exciting, lots of cutting edge cool stuff, and there are people trying to make it easy for beginners to start.

</comment><comment><USER76> Search can cover other domains too... what about an AI that can search books/research articles/lectures/videos to diagnose a medical disease - <USER2> of which aren't actually published live to the internet (perhaps behind paywalls) -- then it takes a person's current symptoms and comes up with the best diagnoses from it's search across multiple media types.

How about search in the context of AR... if people overlay data on top of the world we live in, in AR apps, will there be searchable things there? There's room for search related projects in the future, but it just matters what the data is, and why it's being searched.

Normal search 'engines' for web documents --- that itself seems pretty much 'won' by google, until something better comes along (an implant that has better search than google, and I only need to think about what I want to search for then I automatically download the data to my brain for the top 10 results)

</comment><comment><USER77> Search means a lot of things, but even if we limit to mean web-search, as most people understand it, there is a lot more to it than the actual technology that matches queries to documents.

IR is for all intents and purposes a solved problem -- in fact it was solved a long time ago, and I highly recommend the seminal book "Managing Gigabytes". I also recommend [https://github.com/phaistos-networks/Trinity/wiki/IR- Search-...](https://github.com/phaistos-networks/Trinity/wiki/IR-Search-Links) this page(disclaimer: I am maintaining it) for some interesting/important links to IRC technologies, developments, etc. While some novel ideas come out from time to time, the fundamentals haven't changed -- progress there is incremental and mostly specific to different encoding schemes or ways to execute queries faster by using JIT or more cache-aware datastructures, etc.

Managing and queries documents based on keywords and boolean operators is one thing, and Lucene/Solr, and Trinity ([https://github.com/phaistos- networks/Trinity](https://github.com/phaistos-networks/Trinity)) among other technologies can be used to take care of those challenges. But that's the easy part (assuming you can do this fast enough, because you almost always can't afford long-running queries):

\- User Interfaces: Not just how results are presented, but also how users can construct or input queries. What options can be come available for filtering matches? \- Ranking: Precision is key, and rather simple formulas (tf/idf, BM25, etc) generally don't work well for many/most domains. Furthermore, ranking is almost always not just about relevancy. It factors in static context scores (e.g document "popularity"), personalisation biases(how likely is it for user to mean Soccer or American Football for [football]),and other signals, fused together somehow to determine the final ranking of matched documents. \- Scale: Getting everything right is one thing, getting everything right at massive scale is whole different game. What may work on small scale(algorithms, technologies, services) may not work at all when you scale out. \- Everything else not directly related to search but either important or fundamental to a good experience/business: from matching queries to ads, to analytics, to autosuggestions, to training ML models to power all that, etc.

Web search is not a zero sum game. Bing makes over 3nb / year and while it may not have a chance to catch up with Google anytime soon, that's a great business right there. Ditto for DDG. There are also companies that offer a different or better experience and access to datasets google doesn't yet.

So, all told, search may be solved only in terms of the basic IR technology that makes it all work, and arguably a lot better than it used to be in terms of user interfaces, ranking, etc, but it will take a lot longer until those other aspects of web search may be considered 'solved'.

</comment><comment><USER78> It's definitely not solved from a technical perspective but it's really hard to compete in a business perspective. The market wants good fast search. If you come up with great fast search, it's still hard. The only opportunity I see from the business perspective is to challenge the visual paradigm. Having said that, there are tons of opportunities from the academic perspective such as inferring context, letting users control context, etc.

</comment><comment><USER79> I think Google has pretty much monopolized search. You could say it's solved. I doubt if there are any complaints that sound like 'i couldn't find something using Google'

<comment><USER19> Google ultimately serves the advertisers, not the end users. This incentivizes manipulation of search results in order to maximize ad growth. On that token alone, they have not solved search. Any newcomer developing a search service would do well to make it radically different from Google rather than a clone with added privacy or a Microsoft logo.

Regarding always being able to find what you are looking for on Google: I often struggle to find niche information using Google, and most content on the Web is not indexed by Google.

I'm more of the opinion that YouTube is less possible to compete with in any serious way.</comment></comment>


Chrysaora - WebGL jellyfish - <AUTHOR> http://chrysaora.com/

<comment><USER2> Nice simulation, I really hope IE9 (probably they will adopt it in 10) will adopt the WebGL standard this way we will be able to target a larger audience. Personally I use Chrome so I have no problem coding/using WebGL, however a lot of people I know are still stuck with IE for various reasons.

<comment><AUTHOR> They'll probably come out with WebDX.

</comment></comment><comment><AUTHOR> See the associated blog post at <http://blog.aleksandarrodic.com/?p=33> . There's a fair bit of tech associated with it, including some server-side simulation where the state is passed back to the client via websockets.</comment>