Powerful web scraping framework for Crystal
The CLI has been updated with a few patched bugs. Readme documentation for the CLI has also been added.
Not all links were being scanned because of incorrect keying in the queue
Complete with a full suite of web scraping tools and a cli for some general tasks such as generating a sitemap.xml
The binary requires the following libraries:
libssl-dev
libxml2-dev