Host: libraries.io
User-agent: *
Disallow: /admin
Sitemap: https://libraries.io/sitemaps/sitemap.xml.gz

Host: libraries.io
User-agent: *
Disallow: /admin/
Disallow: /github/
Disallow: /gitlab/
Disallow: /bitbucket/
Disallow: */usage$
Disallow: */dependents$
Sitemap: https://libraries.io/sitemaps/sitemap.xml.gz

Drop some sites from crawlers
Drop some sites from crawlers

Drop repo sites from being crawled (so people's data is less indexed
and hopefully less overhead of people finding them) and also stop
indexing some expensive pages that don't have info that's useful
from an SEO perspective
