I noticed recently that Google is not caching all of the pages on my website. Upon using the Google webmaster diagnostic tool, I realized that some of my pages were being restricted by entries in my robots.txt file.I know some people may frown on the use of wild cards etc, but my web framework generates dynamic URLs - so there is no otherway of 'pattern matching' a set of Urls.I have included both my robots.tx and a list of the ignored URLs here. Hopefully, someone can spot the rule(s) that are restricting Googlebots access to the URLs.First of all, here is a (truncated) list of restricted URLs:URL Detail Detectedhttp://www.example.com/community/groups/cat-lovers-united.html?filter=events URL restricted by robots.txt 10/4/10http://www.example.com/community/knowledge-center/questions/newbie/1/how-to-love-your-cat.html URL restricted by robots.txt 10/4/10http://www.example.com/community/news/comments/1 URL restricted by robots.txt 10/4/10http://www.example.com/feeds/blogposts?request_type=comments&id=2 URL restricted by robots.txt 10/4/10http://www.example.com/news/106/related-headlines?page=1 URL restricted by robots.txt 10/2/10http://www.example.com/news/search?category=agriculturals&keywords=argentina URL restricted by robots.txt 10/6/10http://www.example.com/news/submissions/list?site=ibnlive.in.com URL restricted by robots.txt 9/30/10Here is my robots.txt file:User-Agent: *Disallow: /apiDisallow: /api/Disallow: /logoutDisallow: /logout/Disallow: /loginDisallow: /login/Disallow: /registrationDisallow: /registration/Disallow: /admin/*Disallow: /news/submitDisallow: /*/newDisallow: /*/createnewDisallow: /*/editDisallow: /*/edit/*Disallow: /*/editprofileDisallow: /*/deleteDisallow: /*/delete/*Disallow: /*/updateDisallow: /*/update/Disallow: /*/submitDisallow: /*/submit/Disallow: /*/reportDisallow: /*/report/*Disallow: /searchDisallow: /search/Disallow: /controllers/*Disallow: /community/members/*/joinNetworkDisallow: /community/admin/*Disallow: /lostpasswordDisallow: /lostpassword/Disallow: /registrationCompleteDisallow: /registrationComplete/Disallow: /accountActivateDisallow: /accountActivate/Disallow: /generate-tokenDisallow: /create-widgetDisallow: /buttonsDisallow: /accountDeletedDisallow: /accountSuspendedDisallow: /news/submitDisallow: /news/process-submissionDisallow: /news/submit/successDisallow: /news/errorpageDisallow: /community/members/*/mailinglist/addDisallow: /community/members/*/editprofileDisallow: /community/members/*/joinNetworkDisallow: /community/users/*/followDisallow: /community/users/*/unfollowDisallow: /trustnetwork/createtokenDisallow: /trustnetwork/applytokenDisallow: /community/members/*/mail/*Disallow: /community/mail/*Disallow: /community/blogs/blogposts/unpublished/*Disallow: /community/blogs/blogposts/reported/*Disallow: /tools/pipeline/vote-submitted.htmlDisallow: /community/classifieds/replies/*Disallow: /reportDisallow: /thumbvoteDisallow: /likeDisallow: /dislikeDisallow: /voteRemoveDisallow: /flagAsInterestingDisallow: /addFavoriteDisallow: /recommend/*Disallow: /completed-recommendation.htmlDisallow: /rateDisallow: /notifyDisallow: /addTestimonialDisallow: /comments/update/*Disallow: /comments/great-commentDisallow: /community/members/admin/*Disallow: /community/blogs/blogposts/comments/moderated/*Disallow: /community/blogs/blogposts/comments/reported/*Disallow: /community/articles/reported/fetch/*Disallow: /community/articles/reported/comments/moderated/*Disallow: /community/articles/comments/reported/*Disallow: /community/groups/*/requestsDisallow: /community/groups/*/moderationDisallow: /community/groups/*/headlineMgmtDisallow: /community/groups/*/pollMgmtDisallow: /community/groups/*/joinDisallow: /community/groups/*/leaveDisallow: /community/groups/*/shoutDisallow: /community/groups/*/topics/reply/*Disallow: /community/groups/*/banUserDisallow: /countries/citiesDisallow: /country/schoolsDisallow: /community/knowledge-center/answers/acceptDisallow: /community/knowledge-center/answers/unAcceptDisallow: /requestPasswordDisallow: /restorePasswordDisallow: /restoredPasswordDisallow: /changePasswordDisallow: /changedPasswordDisallow: /adverts/replies/mailboxDisallow: /security/error403Disallow: /home/error404Disallow: /home/disabled-notificationDisallow: /home/error500Disallow: /*?Allow: /Sitemap: http://www.example.com/sitemap-index.xmlIt looks ok to me, but obviously, there must be something in the robots.txt which is restricting access to quite a few pages. I will be very grateful if someone can spot the offending lines in my robots.txt file
Googlebot crawler: URL restricted by robots.txt - how to fix the entries in robots.txt?
google;google search console;robots.txt;web crawlers