A high performance web crawler / scraper in Elixir.
:force
option:scope
optionFull Changelog: https://github.com/fredwu/crawler/compare/v1.4.0...v1.5.0
debug
or warn
Full Changelog: https://github.com/fredwu/crawler/compare/v1.3.0...v1.4.0
:store
option, defaults to nil
to save memory usage:max_pages
optionCrawler.running?/1
to check whether Crawler is runningFull Changelog: https://github.com/fredwu/crawler/compare/v1.2.0...v1.3.0
Crawler.Store.all_urls/0
to find all scraped URLsFull Changelog: https://github.com/fredwu/crawler/compare/v1.1.2...v1.2.0