Butler Versions Save

• Bump up default context deadline to 15 seconds to help with slower connections not allowing builds to be pushed
• Add flag "--context-timeout" for setting the timeout to any value
• Butler is now signed with itch corp. certificate on macOS and Windows

Because of a flawed resume check, the unzip command in v1.0.0 skipped over the first file of the zip archive. v1.0.1 fixes this.

v1.0.0 introduces many new concepts in butler, @elisee asked me what was new so, here it goes:

Manifests (intro)

One big feature is dividing containers in blocks. Up till then we had "archives" (containers compressed as a .zip file), "signatures" (MD5 hashes of <=16KB blocks), and "patches" (list of instructions).

This release introduces a new file format, "manifests", which contains SHAKE256 32-bit hashes of 4MB blocks. They're supported by the "ls" and "file" command, and most of the code writing or reading it is available in the blockpool package of wharf.

There are no new user-facing commands in butler allowing the creation of manifests, this will come in later releases, more details in the later sections.

Healing

Another big feature is healing: given a signature and a heal source, the verify command can now redownload parts of files that were corrupted. It'll also restore missing directories and missing/corrupted symlinks.

Note that when --heal is used, the exit code behavior of verify is modified: it'll now return 0 if it healed everything successfully, whereas without --heal it would've returned 1.

Plans: The apply command will also get a --heal parameter, so that if patch application fails for a reason or another, it'll be able to heal the result, instead of forcing the butler user to fall back to a full re-download.

itchfs

For healing to be interesting, the heal source must be remote — and yet verify's --heal parameter accepts a local path.

However, butler now accepts, for all path arguments, itchfs:///XXX urls. All valid URLs can be devised from the sources.go file of go-itchio.

Note that the API key is passed as a query parameter.

• Example URL: itchfs:///upload/3/download?api_key=XXX

(Yes, that's three slashes - the third is the root, much like file:///etc/hosts)

This means, when an itchfs is passed to the --heal parameter of the verify command, butler will only download the parts of the remote .zip file it needs to heal the local container. Due to the structure of the .zip format, this is relatively efficient:

• First request establishes the size of the file
• Second request to seek to the end and read the Central directory
• Then, reading from any files is just an additional request. If files are read in-order, the same request is re-used throughout.

The itch.io-specific parts are in go-itchio, the io.ReaderAt interface for remote files can be found in httpkit.

Note that the remote server must support the Range HTTP header and respond with a 206 partial content, otherwise an error is thrown.

itchfs is implemented in a pluggable way, which means other vendors are welcome to come up with their own URL schemes and build a version of butler that supports them (for example, for building their own launcher/updater system).

Resumable unzip

The unzip command now accepts a --resume-file parameter. With itchfs URLs, it can be used to download and extract a file at the same time (and resume at the beginning of the last file it was working on, using the resume file's info).

BPS / ETA

All progress-emitting commands now output eta (estimated time remaining, in seconds) in json output mode. They also now emit progress (floating number between 0.0 and 1.0) in addition to percent (floating number between 0.0 and 100.0, kept for backwards compatibility for now).

A few commands (like unzip, cp) are also getting bps (bytes per second processed). This is useful to show download/processing speed graphs:

Manifests (plans)

The big idea for manifests is for them to be usable as a heal source for verify, apply, and as an argument to a new command unsplit. Since manifests are only a list of hashes, access control is left as an exercise to the user — for itch.io, it'll be implemented in the form of download sessions tied to a certain set of blocks, expiring, and started with a valid itch.io API key, which allows for monitoring of abuse and prevents Dropshipping blocks.

Better release support on travis+S3

Auto S3 deployment fiddling

.travis.yml syntax error

Still ironing out travis build

butler knows how to download a single file from an URL to somewhere on the file system.