Ask HN: Is there an industry standard search algorithm for searching websites?
Hello! I was wondering if there is some kind of standard or best use case for searching algorithms for websites. Of course, Googles algorithm is highly secret, and that's absolutely not the scale I'm looking for. I'm talking about simple searches<p>I'm using Sphinxsearch with the BM25 ranker, but the nice part is that I can weight certain fields.
Does somebody have a best use case for indexing and searching websites?<p>For now, I'm indexing websites and save them into a database and set weights on the specific fields<p>100 url - host
90 url - path
80 url
50 title
30 keywords
30 description
20 headings (h1,h2,h3)
10 paragraphs
10 special (strong, blockquotes..)
1 body (text only body)<p>Maybe there is another aspect I didn't think about and maybe there are good resources on the internet that could help out that I missed. I know you can make algorithms insane, but the scale is not so big so I also don't want to overdo it.