Browsertrix Crawler Versions Save

What's Changed

• improved handling of requests from workers: by @ikreymer in https://github.com/webrecorder/browsertrix-crawler/pull/562
• Skip Checking Empty Frame + eval timeout by @ikreymer in https://github.com/webrecorder/browsertrix-crawler/pull/564
• add STORE_REGION env var to be able to specify region by @ikreymer in https://github.com/webrecorder/browsertrix-crawler/pull/565
• PDF loading status code fix by @ikreymer in https://github.com/webrecorder/browsertrix-crawler/pull/571
• Fix regressions with failOnFailedSeed option by @tw4l in https://github.com/webrecorder/browsertrix-crawler/pull/572
• headers: better filtering and encoding by @ikreymer in https://github.com/webrecorder/browsertrix-crawler/pull/573

Full Changelog: https://github.com/webrecorder/browsertrix-crawler/compare/v1.1.1...v1.1.2

What's Changed

• Avoid crashes when editing / creating profile and navigation is interrupted
• profiles: ensure all page.goto() promises have at least catch block/a… by @ikreymer in https://github.com/webrecorder/browsertrix-crawler/pull/559
• profiles: ensure initial page.load() is awaited by @ikreymer in https://github.com/webrecorder/browsertrix-crawler/pull/561

Full Changelog: https://github.com/webrecorder/browsertrix-crawler/compare/v1.1.0...v1.1.1

Major Features

Support for QA Crawling (https://crawler.docs.browsertrix.com/user-guide/qa/)

What's Changed

• QA Crawl Support (Beta) by @ikreymer in https://github.com/webrecorder/browsertrix-crawler/pull/469
• Use RFC2606 invalid domain names by @vnznznz in https://github.com/webrecorder/browsertrix-crawler/pull/514
• Fixes from 1.0.3 release -> main by @ikreymer in https://github.com/webrecorder/browsertrix-crawler/pull/517
• Unify WARC writing + CDXJ indexing into single class by @ikreymer in https://github.com/webrecorder/browsertrix-crawler/pull/507
• upgrade puppeteer-core to 22.6.1 by @ikreymer in https://github.com/webrecorder/browsertrix-crawler/pull/516
• avoid cloudflare detection of puppeteer when using browser profiles: by @ikreymer in https://github.com/webrecorder/browsertrix-crawler/pull/518
• add an extra --postLoadDelay param to specify how many seconds to wait after page-load by @ikreymer in https://github.com/webrecorder/browsertrix-crawler/pull/520
• Gracefully handle non-absolute path for create-login-profile --filename by @tw4l in https://github.com/webrecorder/browsertrix-crawler/pull/521
• Make /app world-readable to better support non-root usage by @vnznznz in https://github.com/webrecorder/browsertrix-crawler/pull/523
• merge V1.0.4 change -> main: by @ikreymer in https://github.com/webrecorder/browsertrix-crawler/pull/527
• Revert "Make /app world-readable to better support non-root usage" by @ikreymer in https://github.com/webrecorder/browsertrix-crawler/pull/529
• ensure all warcwriter write operations go through a queue. by @ikreymer in https://github.com/webrecorder/browsertrix-crawler/pull/528
• qa/replay crawl loading improvements by @ikreymer in https://github.com/webrecorder/browsertrix-crawler/pull/526
• Separate writing pages to pages.jsonl + extraPages.jsonl to use with new py-wacz by @ikreymer in https://github.com/webrecorder/browsertrix-crawler/pull/535
• Adblock support by @ikreymer in https://github.com/webrecorder/browsertrix-crawler/pull/534
• Remove no longer needed invalid Brave update URLs by @tw4l in https://github.com/webrecorder/browsertrix-crawler/pull/539
• Better logging of all queue WARCWriter operations by @ikreymer in https://github.com/webrecorder/browsertrix-crawler/pull/536
• qa: filter out non-html pages by @ikreymer in https://github.com/webrecorder/browsertrix-crawler/pull/541
• Fix for --rolloverSize for individual WARCs in 1.x by @ikreymer in https://github.com/webrecorder/browsertrix-crawler/pull/542
• Set mime type for html pages by @tw4l in https://github.com/webrecorder/browsertrix-crawler/pull/545
• allow minio to connect to other regions by @mguella in https://github.com/webrecorder/browsertrix-crawler/pull/543
• replay counts: don't filter out URLs with __wb_method to avoid dispar… by @ikreymer in https://github.com/webrecorder/browsertrix-crawler/pull/552
• Add crawler QA docs by @tw4l in https://github.com/webrecorder/browsertrix-crawler/pull/551
• Support site-specific wait via browsertrix-behaviors by @ikreymer in https://github.com/webrecorder/browsertrix-crawler/pull/555
• warcinfo: fix version to 1.1 to avoid confusion (part of #553) by @ikreymer in https://github.com/webrecorder/browsertrix-crawler/pull/557

New Contributors

• @mguella made their first contribution in https://github.com/webrecorder/browsertrix-crawler/pull/543

Full Changelog: https://github.com/webrecorder/browsertrix-crawler/compare/v1.0.4...v1.1.0

What's Changed

• Separate writing pages to pages.jsonl + extraPages.jsonl to use with new py-wacz by @ikreymer in https://github.com/webrecorder/browsertrix-crawler/pull/535
• Adblock support by @ikreymer in https://github.com/webrecorder/browsertrix-crawler/pull/534
• Remove no longer needed invalid Brave update URLs by @tw4l in https://github.com/webrecorder/browsertrix-crawler/pull/539
• Better logging of all queue WARCWriter operations by @ikreymer in https://github.com/webrecorder/browsertrix-crawler/pull/536
• qa: filter out non-html pages by @ikreymer in https://github.com/webrecorder/browsertrix-crawler/pull/541
• Fix for --rolloverSize for individual WARCs in 1.x by @ikreymer in https://github.com/webrecorder/browsertrix-crawler/pull/542
• Set mime type for html pages by @tw4l in https://github.com/webrecorder/browsertrix-crawler/pull/545

Full Changelog: https://github.com/webrecorder/browsertrix-crawler/compare/v1.1.0-beta.4...v1.1.0-beta.5

What's Changed

• Gracefully handle non-absolute path for create-login-profile --filename by @tw4l in https://github.com/webrecorder/browsertrix-crawler/pull/521
• refactor handling of max size for html/js/css by @ikreymer in https://github.com/webrecorder/browsertrix-crawler/pull/525
• merge V1.0.4 change -> main: by @ikreymer in https://github.com/webrecorder/browsertrix-crawler/pull/527
• ensure all warcwriter write operations go through a queue. by @ikreymer in https://github.com/webrecorder/browsertrix-crawler/pull/528
• qa/replay crawl loading improvements by @ikreymer in https://github.com/webrecorder/browsertrix-crawler/pull/526

Full Changelog: https://github.com/webrecorder/browsertrix-crawler/compare/v1.1.0-beta.3...v1.1.0-beta.4

What's Changed

• refactor handling of max size for html/js/css by @ikreymer in https://github.com/webrecorder/browsertrix-crawler/pull/525 Fix for #522, issues loading pages with large streaming js/css

Full Changelog: https://github.com/webrecorder/browsertrix-crawler/compare/v1.0.3...v1.0.4

What's Changed

• Use RFC2606 invalid domain names by @vnznznz in https://github.com/webrecorder/browsertrix-crawler/pull/514
• Fixes from 1.0.3 release -> main by @ikreymer in https://github.com/webrecorder/browsertrix-crawler/pull/517
• Unify WARC writing + CDXJ indexing into single class by @ikreymer in https://github.com/webrecorder/browsertrix-crawler/pull/507
• upgrade puppeteer-core to 22.6.1 by @ikreymer in https://github.com/webrecorder/browsertrix-crawler/pull/516
• avoid cloudflare detection of puppeteer when using browser profiles: by @ikreymer in https://github.com/webrecorder/browsertrix-crawler/pull/518
• add an extra --postLoadDelay param to specify how many seconds to wait after page-load by @ikreymer in https://github.com/webrecorder/browsertrix-crawler/pull/520

Full Changelog: https://github.com/webrecorder/browsertrix-crawler/compare/v1.1.0-beta.2...v1.1.0-beta.3

What's Changed

• fixes redirected seed (from #475) being counted againt page limit: by @ikreymer in https://github.com/webrecorder/browsertrix-crawler/pull/509
• sitemap improvements: gz support + application/xml + extraHops fix by @ikreymer in https://github.com/webrecorder/browsertrix-crawler/pull/511

Full Changelog: https://github.com/webrecorder/browsertrix-crawler/compare/v1.0.2...v1.0.3

What's Changed

• Docs: Minor fixes to edit link & clarifications by @Shrinks99 in https://github.com/webrecorder/browsertrix-crawler/pull/501
• Improved support for running as non-root by @ikreymer in https://github.com/webrecorder/browsertrix-crawler/pull/503
• improvements to 'non-graceful' interrupt to ensure WARCs are still closed gracefully by @ikreymer in https://github.com/webrecorder/browsertrix-crawler/pull/504
• service worker capture fix: disable by default for now by @ikreymer in https://github.com/webrecorder/browsertrix-crawler/pull/506
• QA Crawl Support (Beta) by @ikreymer in https://github.com/webrecorder/browsertrix-crawler/pull/469

New Contributors

• @Shrinks99 made their first contribution in https://github.com/webrecorder/browsertrix-crawler/pull/501

Full Changelog: https://github.com/webrecorder/browsertrix-crawler/compare/v1.1.0-beta.1...v1.1.0-beta.2

What's Changed

• service worker capture fix: disable service workers by default for now, add cli option by @ikreymer in https://github.com/webrecorder/browsertrix-crawler/pull/506

Full Changelog: https://github.com/webrecorder/browsertrix-crawler/compare/v1.0.1...v1.0.2