Repeatr Versions Save

Releases for the Timeless Stack as a whole are now bundled together and released on the polydawn/timeless releases page: https://github.com/polydawn/timeless/releases/tag/release%2Fv0.200rc1

Portability and internal systems updates galore: v0.15 should quietly make things better. No major shakeups to formulas or APIs.

Have you seen the new projects in the ecosystem? Reppl brings pipelining tools to the table, so you can now compose complex build processes with multiple formulas easily. r2k8s demonstrates integrating Repeatr and Kubernetes, making it possible to run services in k8s, as well as use k8s clusters as resources for running builds. Repeatr might not be changing much in this release, but there's a lot of other stuff to play with!

Full list of changes:

• Change: runc executor updated! Repeatr now references a runc release from the 1.0rc3+ region (build specified in a formula, and reproducible).
• Change: the working directory will now always be ensured to be owned and writable by the container process UID. (This is followup to the "cradle" system introduced back in v0.11, and if necessary can be disabled the same way as before.)
  • This makes it much easier to have inputs that are placed inside the working directory! Previously that would cause directories to be implicitly created as necessary, potentially resulting in a CWD that was unwritable to UID>0. Now, such input configuration may still result in such implicit directory creation, but the CWD itself will consistently be writable, which is usually what the user expects.
• Feature: overlay is now supported as a COW filesystem! If your host has overlayfs, it will be used in preference to AUFS; then AUFS, then so on down the fallback tree as before.
  • Note that there is no "migration" necessary, because why would there be?
• Bugfix: now emit well-formed tar when an output is a single file. Previously the tar emitted would always mark the first entry as a dir, and thus not be valid if the following content was a single file.
• Bugfix: if your system lacks a modprobe command, handle this gracefully.
• Bugfix: additional fatal errors from runc are now correctly raised.
• Bugfix: repeatr unpack now overwrites destination locations if they already exist, rather than erroring -- as documented.
• Feature: the filesystem assembly now supports files as well as directories -- meaning the "file" transmat can be used in formulas, as can bind mounts of individual files from the host.

v0.14's most notable change is a whole new suite of integration opportunities: a JSON API to repeatr run now allows you to handle logs, results, and job output clearly using external tools.

The git transmat has been massively upgraded, and now handles deduplication and submodules much, much better. The improvements should be huge for anyone using git, especially with slow remotes, large repos, or many submodules.

Full list of changes:

• Change: repeatr scan is now known as repeatr pack, because that's a much more accurate description of what it does.
• Bugfix: Results in a RunRecord are serialized in a consistent order again as they should be. This was a regression introduced when we moved from "outputs" to the slightly terser "results" structs.
• Feature: Errors got a facelift and consistency rework. Many errors are now reported more tersely and helpfully at the command line. And errors in the API are properly serializable.
  • All of these new error types are exported in the def package and are easily serializable, with strongly-typed fields.
  • If you're using the API client packages in repeatr//api/act/remote, yes -- you will get ErrHashMismatch and friends to use programmatically, complete with all of their detail fields.
• Internal: Package dependencies for the api/* packages are now validated in CI. This helps us make sure we don't bloat the dependencies of the API packages accidentally.
• Feature: API-ready machine-parsable event and log streams!
  • Use the new --serialize flag to repeatr run to enable this feature.
  • All output will be formatted as json messages, one per line, easy to parse. (CBOR support will be coming in the future for more efficient but less human readable operation.)
  • Check out the new repeatr//api/act/remote package for a client implementation that can be easily imported into any other golang programs! Like the rest of the api package, this has no direct dependencies on the rest of repeatr (e.g. you won't get container engines in your dependency tree; just API, as it should be).
• Feature: Massive overhaul to the git transmat, which should massively increase efficiency and successful caching.
  • Submodules are now cached: if you have a parent project move to a new commit, but keep all the same submodule versions, previously repeatr was oblivious; now, it's a fast 100% cache hit.
  • Git data objects are now cached: fetches are incremental when using the same remote repos.
  • Slight change: ".git" files in submodule paths no longer leak into the container's sight. This shouldn't affect you unless you had output configurations exporting filesystems including such files.
• Feature: repeatr run now has a --serialize (or -s) flag that serializes all output and writes it to stdout.
• Internal: Added "exercise" script, which takes a final repeatr binary through the full cycle of major commands; useful for full validation on host environments.
• Bugfix: Respect output filter configuration properly again! This was broken in v0.12 and v0.13.
• Bugfix: Bubble up errors from the AUFS placer correctly when mount fails!
• Feature: repeatr unpack now accepts a --skip-exists flag, which will skip unpacking if the target path already exists. This does not check that the path matches the hash given to the unpack command; be careful when using this.
• Feature: repeatr unpack now works atomically, using tempdirs (or tempfiles) in the target directory.
• Bugfix: Repeatr OSX builds fixed. You can now use repeatr unpack on OSX! (As long as you don't ask for any data that can't be losslessly expressed on a mac -- namely, filesystems containing symlinks still error, because mtimes cannot be set with full precision.)
• Feature: A single file transmat may now be used with the repeatr unpack command to get data that is a single file with no filesystem metadata.
• Bugfix: Handle error codes the same for http and https tranports.
• Bugfix: Version info string no longer includes double "v" typo.

v0.13 is calm sailing: we have a variety of performance improvements, a few new config options, several logging improvements, and essentially no major shockers.

If parsing the output struct of repeatr run, note the changes to format there. (We made things a great deal terser; also, "outputs" was renamed to "results".)

Full list of changes:

• Internal: Major refactor to package structure. IO components now more clearly separated from sandbox/execution and other bits of repeatr core.
• Internal: Types now gathered under api/* packages, so that these can be easily linked to help integrate other systems with repeatr.
• Bugfix: Several error handling paths in the tar transmat are now considered WarehouseUnavailableError (instead of the more red-flaggy WarehouseIOError), allowing other sources to be tried.
• Feature: Formulas now accept an action.hostname parameter, going along with other environmental specifiers there. This will set the hostname (in execution engines that support this feature).
• Internal: Scheduler package removed.
• Improvement: Assets used for testing now have a more scripted process for priming your cache for offline work.
• Improvement: More logging during filesystem setup before execution and the scanning afterward. Now includes easy to see $n/$m progress reporting.
• Internal: Executors now use take a structured logger as a parameter. No more surprisingly deeply-reaching byte streams.
• Internal: Formula execution implemented through the new api interfaces.
• Change: The output of repeatr run is a new structure, and less verbose again (it only speaks of results, and doesn't repeat the entire formula).
• Feature: repeatr twerk can now accept several kinds of formula patches -- policy settings, env vars, etc.
• Feature: repeatr cfg subcommand now exists to make your life easier: if you want yaml formulas transformed into easier-to-handle json, you got it.
• Internal: The root package domain was changed.
• Improvement: Several transmats compare hashes in a slightly more user-friendly way (specifically, they b64 things, then compare, rather than the other way around -- this results in a more helpful error message in case your formula contained a typo in the hash that doesn't parse as b64).
• Internal: The def/api package no longer has an external dependency on an unusual error handling library. This should make it much easier to import in other go projects.
• Improvement: Reduced CPU spend on output streaming. (Buffer jitter may have increased, but shouldn't be particularly perceptible in practical use.)

The major highlights of v0.12 are some improved flexibility in commands (you can now apply "patches" to formulas for quick-n-easy configuration), and performance improvements (compression is finally enabled for most storage).

The default executor for repeatr run is now the 'runc' system -- meaning fine-grained capabilities and security features from the Policy system introduced in v0.11 will now be impactful in the default modes.

Also, this is the first version of Repeatr built with the recently-released go 1.6! This should result in all-around performance improvements due to continuing upstream improvements to garbage collection (and some optimizations to sorting which will probably make a significant dent in our filesystem hashing as well).

Full list of notable changes:

• Feature: repeatr run now outputs the exit code in the structure it sends to stdout, so it can be mechanically extracted and clearly disambiguated from repeatr's own exit code.
• Feature: repeatr run now accepts additional snippets of partial formulas with the -p flag, and will patch them onto its main argument. This allows simple scripts to provide custom values to a run without needed to sprout a whole json/yaml parser. The fully patched values will appear in the formula emitted at the end of the run along with the output hashes, as you might expect. Use judiciously; this functionality makes sense in repeatr run since one-off runs are its MO, but not all upcoming features will support this particular escape valve (in particular, pipelines certainly won't). Currently only env vars are merged.
• Feature repeatr run now accepts env vars with the -e flag, and will patch them onto the formula. This is shorthand for doing the same with -p.
• Bugfix: When using 'tar' transports with 'http' or 'https' URLs, HTTP status codes of 404 will now be reported as DataDNE errors. Previously, this would be incorrectly reported as existing but corrupt data.
• Change: repeatr run now accepts the formula file as a positional argument (you can get rid of the -i in all your scripts).
• Change: The default executor in repeatr run is now 'runc' instead of 'chroot'. (You can continue to use the chroot executor by flagging --executor chroot.)
• Improvement: All transports which store filesystems in tar format (this includes 'tar', 's3' and 'gc') now upload gzip compressed data by default.
• Bugfix: When using the 'git' transport, relative paths like ".." are now handled much more reasonably. (Internally, all paths are absolutized by repeatr before invoking git, to work around a series of very interesting git behaviors; however, since git metadata is already not exposed to the filesystem inside containment, this change should be quite transparent.)
• Change: repeatr twerk default image updated. As always, remember: you're not actually supposed to use this feature outside of experimentation and feel embarrassed if you do; changes are no-warning.
• Bugfix: Subcommands with incorrect usage now exit with a status code of 1 after printing their help text. (Previously, they incorrectly exited with a zero!)

This release of Repeatr includes the "Policy" system -- this is majorly exciting: for the first time, we have containers which drastically reduce the privilege of processes inside them by default. This is a major improvement to security for users, and hopefully the start of major improvements to the whole ecosystem, since safe operations are now the default operations. Of course, it's also a massively breaking change for any formulas that previously required powerful and unsafe system permissions -- they now have to admit it up-front! ;)

• Feature: Policies!! And graceful de-escalation of privileges. [PR: gh#68]
  • By default, executors will drop to user-level privileges and a non-0 (a.k.a non-root) UID.
  • Executors which support advanced features like linux capabilities will also drop those.
  • Policy levels available are, from safest to most empowered: routine, uidzero, governor, and sysad. Routine is the default.
• Feature: Several minimum-viable-provisioning will be applied to your filesystems and environment before job launch: this is called the "cradle". These features make operating with low privileges (as introduced concurrently by the policies feature) much easier.
  • If you configure a cwd that doesn't already exist, it will be automatically created and be writable.
  • Your jobs may now reliably expect /tmp to exist and to be writable (specifically, it will be forced to chmod=01777; world-writable plus sticky bit, as a tempdir should be).
  • The $HOME environment variable will now be assigned by default. The referenced directory will exist (and be writable, if cradle created it).
  • These new behaviors can be disabled by configuring action.cradle = false in your formulas.
• Bugfix: Clean up the filesystem more gingerly if major errors are raised during executor operation. Certain failure cases of unmounting could previously cause more files to be removed during "cleanup" -- if you're using host mounts, this could be a fairly major problem and you should upgrade immediately.
• Bugfix: Files produced by the 'git' transport will now be owned by uid=1000, gid=1000. This is consistent with the default filter values for other transports.
• Internal: Defining a mechanism to feed results of one formula into another, describing ways to communicate well-known ware hashes by name, and thereupon build automatic update systems and complex processing pipelines. Proof-of-concept work -- will not be externally exposed or API-stable for some time. [PR: gh#67]

A perfectly unremarkable v0.x release.

List of changes:

• Feature: The repeatr twerk subcommand will now mount your current host directory into the container. Writably. Remember: this feature is for exploration and play; it is not the paragon of safe defaults.
• Bugfix: The repeatr twerk subcommand now exits non-zero/failure if the contained process itself exits non-zero/failure. This is consistent with the behavior of repeatr run.
• Bugfix: Try harder to enable AUFS: Repeatr will now attempt to load the kernel module if it's installed but not loaded.