▁advis ing ▁rob ots ▁not ▁to ▁index ▁a ▁very ▁large ▁database , ▁web ▁pages ▁that ▁are ▁very ▁trans itory , ▁web ▁pages ▁that ▁are ▁under ▁development , ▁web ▁pages ▁that ▁one ▁wishes ▁to ▁keep ▁slightly ▁more ▁private , ▁or ▁the ▁printer ▁and ▁mobile - friend ly ▁versions ▁of ▁pages . ▁Since ▁the ▁bur den ▁of ▁hon oring ▁a ▁website ' s ▁no index ▁tag ▁lies ▁with ▁the ▁author ▁of ▁the ▁search ▁robot , ▁sometimes ▁these ▁tags ▁are ▁ignored . ▁Also ▁the ▁interpretation ▁of ▁the ▁no index ▁tag ▁is ▁sometimes ▁slightly ▁different ▁from ▁one ▁search ▁engine ▁company ▁to ▁the ▁next . ▁ ▁No index ing ▁entire ▁pages ▁▁ ▁< html > ▁< head > ▁▁ ▁< meta ▁name =" rob ots " ▁content =" no index "> ▁▁ ▁< title > Don ' t ▁index ▁this ▁page </ title > ▁</ head > ▁ ▁Possible ▁values ▁for ▁the ▁meta ▁tag ▁content ▁are : ▁" none ", ▁" all ", ▁" index ", ▁" no index ", ▁" nofollow ", ▁and ▁" follow ". ▁A ▁combination ▁of ▁the ▁values ▁is ▁also ▁possible , ▁for ▁example : ▁< meta ▁name =" rob ots " ▁content =" no index , ▁follow "> ▁ ▁Bot - specific ▁direct ives ▁ ▁The ▁no index ▁directive ▁can ▁be ▁restricted ▁only ▁to ▁certain ▁b ots ▁by ▁specifying ▁a ▁different ▁" name " ▁value ▁in ▁the ▁meta ▁tag . ▁ ▁For ▁example , ▁to ▁specifically ▁block ▁Google ' s ▁bot , ▁specify : ▁ ▁< meta ▁name =" google bot " ▁content