Using robots.txt I am attempting to stop all crawling of search URLsDisallow: /rest_of_url/search&tour*Above is what I am using. Our URL looks like the following for all search results. However, everything after search&tour can be different, for example:https://www.example.com.au/rest_of_url/search&tour-sdfs=the-palce+lcation+&tour-duration=1/Will the Disallow code above stop robots from crawling all of my search result pages?
Disallow Crawling of All Search Pages
google;google search console;web crawlers;robots.txt;googlebot
Will the Disallow code above stop robots from crawling all of my search result pages?Yes, it will stop the (good) bots that obey the robots.txt standard.However, you don't need the trailing *. robots.txt is prefix matching, so the wildcard * at the end can simply be omitted. (Wildcard type matches are an extension of the original standard anyway.)And you obviously need the User-agent directive that precedes this rule, if you haven't got it already:User-agent: *Disallow: /rest_of_url/search&tour