Bandersnatch Versions Save

New Features

• Add support of downloading packages from alternative mirror PR #928 -- Thanks happyaron
• Add support of s3 storage PR #886 -- Thanks LeoQuote

Bug fixes

• Add bandersnatch command line help to the documentation main page PR #920 - Thanks ichard26
• Generate data-yanked tag in simple page PR #931 - Thanks happyaron
• Protect repository metadata from being trashed when disk is full PR #962 - Thanks happyaron
• Fix tox to used pinned requirements*.txt files for deps - PR #1011 - Thanks cooperlees

Documentation

• Add Storage Options documentation PR #1006 - Thanks cooperlees
• Update all dependencies to latest except pyparsing + docutils

Dependencies

• Change pyup to run monthly + auto add skip_news PR #1043 - Thanks cooperlees

New Features

• bandersnatch is now a >= 3.8 Python project
• New size_project_metadata filter plugin, which can deny download of projects larger than defined threshold - PR #806
• Add option to compare file size and upload time instead of sha256sum for downloading - PR #822
• Add optional uvloop support - PR #891 - Thanks cooperlees
• Move to official docker upload action w/arm64 images uploaded - PR #896 - Thanks cooperlees

Deprecations

• blacklist/whitelist will no longer work in bandersnatch configuration - PR #897 - Thanks cooperlees
  • Please use allowlist/denylist respectively

Bug Fixes

• Unused storage plugins are loaded and cause non-fatal errors if dependencies are missing - PR #799 - Thanks electricworry
• Replaced usages of asynctest with unittest.mock in tests - PR #807 and PR #856 - Thanks ichard26
• Remove debugging line that loads entire files into memory. - PR #858 - Thanks asrp
• Removed terrible isinstance check of unittest.Mock in mirror.py - PR #859 - Thanks ichard26
• Put potential time consuming IO operations into executor - PR #877
• Migrated Markdown documentation from recommonmark to MyST-Parser + docs config clean up - PR #879 - Thanks ichard26
• Use shutil.move() for temp file management - PR #883 - Thanks happyaron
• Fixed logging bug in SizeProjectMetadataFilter to show it activated - PR #889 - Thanks cooperlees
• Attempt to wrap all potentially block calls in a ThreadPoolExecutor - PR #894 - Thanks cooperlees

New Features

• Build a swift and non swift docker image - PR #754
• Split Docker Build to accept build args to optionally include swift support - PR #741 - Thanks nlaurance-pyie
• Slimmer docker image - PR #738 - Thanks nlaurance-pyie
• Renamed black/white to block/allow lists - PR #737 - Thanks nlaurance-pyie
• packages allowlist can be defined from requirements like files - PR #739 - Thanks nlaurance-pyie
• Simplify logging around filters - PR #678 - Thanks @dalley

Bug Fixes

• Handling of timeouts that can occur in verify. - PR #785 - Thanks electricworry
• Added retry logic on timeouts when fetching metadata - PR #773 - Thanks gerrod3
• Fix links, improve docs CI, and improve external object linking - PR #776 - Thanks ichard26
• Handle 404 status for json verify - PR #763 - Thanks electricworry
• Clean up isort config after upgrade to 5+ - PR #767 - Thanks ichard26
• Remove duplicate max() target serial finding code + update typing - PR #745
• swift.py: use BaseFileLock's lock_file property - PR #699 - Thanks hauntsaninja
• Move to latest isort + mypy fixes - PR #706
• Update change log url in project metadata - PR #673 - Thanks @abn

New Features

• Add SOCKS proxy support to aiohttp via aiohttp-socks - PR #668
• Add support for skipping mirroring release files (metadata only) - PR #670 - Thanks @abn

Bug Fixes

• Move GitHub actions to v2 tags - PR #666 - Thanks @ryuichi1208

New Features

Thanks to RedHat engineers @dalley + @gerrod3 for all this refactor work in PR #591

• New generic Mirror class to perform Python metadata syncing
  • (previous Mirror class has been renamed to BandersnatchMirror)
• Package's filter methods are now part of its public API
• New errors.py file to house Bandersnatch specific errors

Internal API Changes

• Old Mirror class has been renamed to BandersnatchMirror. Performs same functionality with use of new Mirror API.
• BandersnatchMirror now performs all filesystem operations throughout the sync process including the ones previously in Package.
• Package no longer performs filesystem operations. Properties json_file, json_pypi_symlink, simple_directory and methods save_json_metadata, sync_release_files, gen_data_requires_python, generate_simple_page, sync_simple_page, _save_simple_page_version, _prepare_versions_path, _file_url_to_local_url, _file_url_to_local_path, download_file have all been moved into BandersnatchMirror. Package's sync has been refactored into Bandersnatch's process_package.
• Package class is no longer created with an instance of Mirror
• StaleMetadata exception has been moved to new errors.py file
• PackageNotFound exception has been moved to new errors.py file

Bug Fixes

• Fixed Fix latest_release plugin to ensure latest version is included - PR #660 - Thanks @serverwentdown

• Fixed name parsing issue for allow/blocklist project filters - PR #651 - Thanks @gerrod3

New Features

• bandersnatch is now 100% type annotated - PRs #546 #561 #592 #593 - Thanks @ichard26 + @rkm
• Move to storage abstraction - PR #445 - Thanks @techalchemy
  • Can now support more than just filesystem e.g. swift
• Add sync subcommand to force a sync on a particular PyPI package - PR #572 - Thanks @z4yx
• Added new allowlist filter - PR #626 - Thanks @gerrod3
• Make webdir/pypi/json/PKG symlinks relative - PR #637 - Thanks @indrat
  • Makes mirror files more portable
• Add main and program name override to ArgumentParser - PR #643 - Thanks @rkm
  • Allow non pkg_resources install to work

Internal API Changes

• Refactored the removal of releases for release_plugins to happen inside of Package PR #608 - Thanks @gerrod3
• Minor refactor of Package class PR #606 - Thanks @dralley
• Refactored filter loading into seperate class PR #599 - Thanks @gerrod3
• Move legacy directory cleanup to mirror.py PR #586
• Move verify to use Master for HTTP calls - PR #555
• Move http request code for package metadata to master.py - PRs #550 - Thanks @dralley

Bug Fixes

• Fixed allow/blocklist release filtering pre-releases - PR #641 - Thanks @gerrod3
• Casefold (normalize per PEP503) package names in blacklist/whitelist plugins config - PR #629 - Thanks @lepaperwan
• Fix passing package info to filters in verify action. PR #638 - Thanks @indrat
• Fix todo file removal - PR #571
• Introduce a new global-timeout config option for aiohttp coroutines - Default 5 hours - PR #540 - Thanks @techalchemy
• Many doc fixes - PRs #542 #551 #557 #605 #628 #630 - Thanks @pgrimaud + @ichard26 + @hugovk
• Move to setting timeout only on session + 10 * total_timeout (over sock timeouts) - PR #535
• Stop using include_package_data option in setup.cfg to get config files included in more installs - PR #519

• Change aiohttp-xmlrpc to use Master.session to ensure config shared - PR #506 - Thanks @alebourdoulous for reporting
  • e.g. Maintin trust of proxy server environment variables

• Raise for error HTML response on all aiohttp session requests - PR #494 / #496 - Thanks @windtail
• Pass str to shutil.move due to Python bug - PR #497 - Thanks @SanketDG
• Some more type hints added to verify.py - PR #488 - Thanks @SanketDG
• Ignore atime on stat in test test_package_sync_does_not_touch_existing_local_file comparision as it casues stat compare fail on a slower run - PR #487 - Thanks @SanketDG

• Pass correct aiohttp timeout objects - PR #478
• Replace pkg_resources with importlib.resources - PR #479 - Thanks @SanketDG