Fredwu Crawler Versions Save

A high performance web crawler / scraper in Elixir.

v1.5.0

5 months ago
  • [Added] Add :force option
  • [Added] Add :scope option

Full Changelog: https://github.com/fredwu/crawler/compare/v1.4.0...v1.5.0

v1.4.0

5 months ago
  • [Added] Allow multiple instances of Crawler sharing the same queue
  • [Improved] Logger will now log entries as debug or warn

Full Changelog: https://github.com/fredwu/crawler/compare/v1.3.0...v1.4.0

v1.3.0

5 months ago
  • [Added] :store option, defaults to nil to save memory usage
  • [Added] :max_pages option
  • [Added] Crawler.running?/1 to check whether Crawler is running
  • [Improved] The queue is being supervised now

Full Changelog: https://github.com/fredwu/crawler/compare/v1.2.0...v1.3.0

v1.2.0

5 months ago

v1.2.0 [2023-09-29]

  • [Added] Crawler.Store.all_urls/0 to find all scraped URLs
  • [Improved] Memory usage optimisations

Full Changelog: https://github.com/fredwu/crawler/compare/v1.1.2...v1.2.0

v1.1.2

2 years ago

v1.1.1

3 years ago

v1.1.0

5 years ago