Recently, I have enabled all my servers to serve everything over HTTP and HTTPS. Users can access any site via http://www.example.com or https://www.example.com. All pages are identical between the versions, so http://www.example.com/about.php is the same as https://www.example.com/about.php and so on.URLs are relative, so they do not mention the protocol with one exception. In other words, if the page is loaded with HTTP, it will link to other pages, images, CSS, Javascript over HTTP and the same with HTTPS, as to avoid mixed content warnings.Now about that exception. It is in robots.txt:Sitemap: http://www.example.com/sitemap.phpApparently this URL must be absolute.Now the problem I see if that when Google reads https://www.example.com/robots.txt it gets an HTTP sitemap! The documentation on robots.org says that one can specify multiple sitemaps but if I am not sure that putting both the HTTP and HTTPS sitemap is a good idea since they will contain each a list of identical pages (one with HTTP and one with HTTPS).How should Sitemap in robots.txt be handled for websites that accept  HTTP and HTTPS?Some ideas that came to mind:Specify both sitemaps (as mentioned above). Afraid this would cause duplicate content issues.Only specify the HTTPS Sitemap. That gives access to all unique pages anyway.Find a magical (Apache) way to sent a different robots.txt via HTTP and HTTPS. Is that even possible? Could it cause issues?
Protocol Agnostic Robots Sitemap
sitemap;robots.txt
A sitemap at http://www.example.com/sitemap.php can only contain URLs from http://www.example.com/. The scheme and the host must be the same.So if you 1) want to provide sitemaps for both protocols, and 2) link both sitemaps via the Sitemap field in the robots.txt, you have to provide separate robots.txt files for HTTP and HTTPS:#        http://www.example.com/robots.txtSitemap: http://www.example.com/sitemap.php#        https://www.example.com/robots.txtSitemap: https://www.example.com/sitemap.php(It should be easy to achieve this with Apache, see for example the answers to Is there a way to disallow crawling of only HTTPS in robots.txt?)But you might want to provide a sitemap only for the canonical variant (e.g., only for HTTPS), because there is not much point in letting search engines parse the sitemap for the non-canonical variant, as they typically wouldnt want to index any of its URLs. So if HTTPS should be canonical:On each HTTP page, link to its HTTPS version with the canonical link type.Provide a sitemap only on HTTPS, listing only the HTTPS URLs.Link the sitemap (ideally only) from the HTTPS robots.txt. Except if cross submits are used.