A simple but powerful web crawler library for .NET
@Turnerj, @dependabot and @dependabot[bot]
Improved support for in-page Robots configuration (#7) Added fragment support for crawl data (#7) Added support for request timeouts (#7) Added support for max pages to crawl (#7) Obey no-follow on anchors (#7) Various other minor tweaks and fixes (#7)
Replaced crawl contents stream with a string, allowing easier access to contents.
Initial release with core functionality