Headless Chrome Crawler Versions Save

Added

• Support obeyRobotsTxt for crawler.queue()'s options.
• Support persist for RedisCache's constructing options.

changed

• Make cache to be required for HCCrawler.connect() and HCCrawler.launch()'s options.
• Provide skipDuplicates to remember and skip duplicate URLs, instead of passing null to cache option.
• Modify BaseCache interface.

Added

• Support CSV and JSON Lines formats for exporting results
• Emit requeststarted, requestskipped, requestfinished, requestfailed, maxdepthreached, maxrequestreached and disconnected events.
• Improve debug logs by tracing public APIs and events.

Changed

• Allow onSuccess and evaluatePage options as null.
• Change crawler.isPaused, crawler.queueSize, crawler.pendingQueueSize and crawler.requestedCount from read-only properties to methods.

Fixed

• Fix a bug of ignoring maxDepth option.

[1.2.3] - 2017-12-17

changed

• Refactor by changing tye style of requiring cache directory.

Fixed

• Fix a bug of starting too many crawlers more than maxConcurrency when requests fail.

Added

• Automatically collect and follow links found in the requested page.
• Support maxDepth for crawler.queue()'s options.

Added

• Support screenshot for crawler.queue()'s options.

Changed

• Rename ensureCacheClear to persistCache for HCCrawler.connect([options]) and HCCrawler.launch([options])'s options

Added

• Support maxRequest, allowedDomains and userAgent option for crawler.queue([options])
• Support pluggable cache
• Add crawler.setMaxRequest(maxRequest), crawler.pause() and crawler.resume() methods
• Add crawler.pendingQueueSize and crawler.requestedCount read-only properties

[1.1.1] - 2017-12-09

Added

• Add CHANGELOG
• Automatically dismisses dialog
• Enrich unit tests

Changed

• Refactor by separating HCCrawler and Crawler classes
• Make preparation of pages parallel

• Refactor by separating HCCrawler and Crawler classes
• Public API to launch a browser has changed. Now you can launch browser by HCCrawler.launch()
• Rename shouldRequest to preRequest
• Modify README according to new public API
• Modify examples according to new public API
• Support extraHeaders option
• Refactor handlers for options
• Add comment in JSDoc style

• Add test with mocha and power-assert
• Add coverage with istanbul
• Add setting for CircleCI
• Add .editorconfig
• Migrate from NPM to Yarn
• Refactor helper to class static method style
• Add debug log