I have full text search engine installed on my site, accessible from URL like /search/<search-query>, and search landing in /search, containing large Search button and some popular tags. Do I need to hide these pages from search bots using robots.txt Disallow entries?
Hiding search pages via robots.txt
robots.txt;search
Indexing search result pages isn't always the best idea - check this article from Matt Cuttsquote: Typically, web search results dont add value to users, and since ourcore goal is to provide the best search results possible, we generallyexclude search results from our web search index.It's quite possible that your search result pages do offer added value for visitors - so difficult to judge whether or not you should block them in the robots.txt. If it's just a list with articles on the same topic (similar to /tag/ pages) - I would consider to put them on noindex / follow.