Exclude domain in robots.txt
Robots.txt will not help your site being crawled (technically) by anything. They're just guidelines which Bots should follow. You should limit dev and test server access by Firewall rules, or as you've mentioned by httpauth.