File robots.txt needed in Craft package? I dont understand why we need the robots.txtfile in the Craft CMS package. I think it is confusing. (I had a problem with Google indexing one of my BETA hidden website because of that.)

Reason : When I initiate my domain name on the hosting it creates automatically a public folder inside the virtual host main folder.  

So for example my hosting server make the domain name point to /lamp0/web/vhosts/public/. So I just have to upload craft and public folders below in /lamp0/web/vhosts/.
Then I just have to rename public in htdocs and no need of robots.txt file because craft folder is below domain name root.



# Don't allow web crawlers to index Craft
User-agent: *
Disallow: /craft/


=========================
UPDATE
OK I see now in the Craft install documentation (see below). I missed it sorry.
Perhaps would it be good to insert a README.txt file in the package next to robots.txt file to explain that again :)
The specific directive to disallow the /craft/ directory isn't necessary if your craft folder is below the webroot.

However, it's good practice to have a robots.txt for each site, so I don't recommend deleting it altogether. You can change the directives to fit whatever robots criteria you need for your site. (For example, if you launch a beta site, you'd want to Disallow the whole thing to avoid duplicate content penalties.)