title: Ignoring robots.txt and meta tags in a crawler
body: <p>Are there a way to make a web crawler ignore the robots.txt file and  tags? Yes, I know this could come with legal repercussions. This question is much like <a href="https://stackoverflow.com/questions/8386481/web-crawler-ignore-robots-txt-file">another question</a> but the answers were very vague and I didn't quite get it. any help is appreciated.</p>