Is there more information available about the web crawler technology/engine used by Kentico 10 as per documentation [Configuring Page Crawler Indexes](https://docs.kentico.com/k10/configuring-kentico/setting-up-search-on-your-website/creating-search-indexes/defining-page-indexes#Definingpageindexes-Configuringpagecrawlerindexes)? 

The reason I'm asking is because I'd like to consider it for use in a custom crawler project that can sit outside of Kentico, and still allow for it to have an inherent compatibility with the Kentico platform.