Quilt Versions Save

Python API

• [Performance] 2X to 5X faster multi-threaded hashing of S3 objects (#1816, #1788)
• [Fixed] Bump minimum required version of tqdm. Fixes a crash (UnseekableStreamError) during upload retry. (#1853)

CLI

• [Added] Add --meta argument to push (#1793)
• [Fixed] Fix crash in list-packages (#1852)

Python:

• Refactors local and s3 storage-layer code around a new PackageRegistry base class (to support improved file layouts in future releases)
• Multithreaded download for large files, large performance gains when installing packages with large files, especially on large instances
• Package name added to Package.resolve_hash
• Bugfix: remove package revision by shorthash
• Performance improvements for build and push

Catalog & Lambdas:

• PDF previews
• Browse full package contents (no longer limited to 1000 files)
• Indexing and search package-level metadata
• Fixed issue with download button for certain text files
• FCS files: content indexing and preview
• Catalog sign-in with email (or username)
• Catalog support for sign-in with Okta

Catalog

• .cef preview
• allow hiding download button
• only show stats for 2-level extensions for .gz files

Python

• quilt3.logged_in()
• fix retries during hashing
• improve progress bars
• fix quilt3 catalog
• expanded documentation
• reduce pyyaml requirements to prevent version conflicts

Backend

• improve unit test coverage for indexing lambdas
• fix real-time delete handling (incl. for unversioned objects)
• handle all s3:ObjectCreated: and ObjectRemoved: events (fixes ES search state and bucket Overview)

Python API

• Official support for Windows
• Add support for Python 3.7, 3.8
• Fix Package import in Python
• Updated libraries for stability and security
• Quiet TQDM for log files ($ export QUILT_MINIMIZE_STDOUT=true )
• CLI setting of config parameters

Catalog

• new feature to filter large S3 directories with regex
• more reliable bucket region inference
• Support preview of larger Jupyter notebooks in S3 (via transparent GZIP)
• JS (catalog) dependencies for stability and security
• extended Parquet file support (for files without a .parquet extension)
• Improvements to catalog signing logic for external and in-stack buckets

Special thanks to @NathanDeMaria (CLI and Windows support) and @JacksonMaxfield for contributing code to this release.

Python

• Add push to CLI

Catalog

• Updated JS dependencies
• Display package truncation warning in Packages

Python

• quilt3 install foo/bar/subdirectory
• Bug fixes for CopyObject and other exceptions

Python Client

• Fix bug introduced in 3.1.9 where uploads fail due to incorrect error checking after a HEAD request to see if an object already exists (#1512)

Python Client

• quilt3 install now displays the tophash of the installed package (#1461)
• Added quilt3 --version (#1495)
• Added quilt3 disable-telemetry CLI command (#1496)
• CLI command to launch catalog directly to file viewer - quilt3 catalog $S3_URL (#1470, #1487)
• No longer run local container for quilt3 catalog (#1504). See (#1468, #1483, #1482) for various bugs leading to this decision.
• Add PhysicalKey class to abstract away local files vs unversioned s3 object vs versioned s3 object (#1456, #1473, #1478)
• Changed cache directory location (#1466)
• More informative progress bars (#1506)
• Improve support for downloading from public buckets (#1503)
• Always disable telemetry during tests (#1494)
• Bug fix: prevent misleading CLI argument abbreviations (#1481) such as --to referring to --tophash
• Bug fix: background upload/download threads are now killed if the main thread is interrupted (#1486)
• Performance improvements: load JSONL manifest faster (#1480)
• Performance improvement: If there is an error when copying files, fail quickly (#1488)

Catalog

• Better package listing UX (#1462)
• Improve bucket stats visualization when there are many categories (#1469)

Python API

• Bug-fixes for quilt3.config and quilt3.catalog
• Performance improvements for Packages

Catalog

• Updated landing page

Catalog

• New LOCAL mode for running the catalog on localhost

Python API

• quilt3 catalog command to run the Quilt catalog on your local machine
• quilt3 verify compares the state of a directory to the contents of a package version
• Added a local file cache for installed packages
• Performance improvements for upload and download
• Support for short hashes to identify package versions
• Adding telemetry for API calls