Ketrew Versions Save

This is a maintenance release the code and the build have been updated to the recent versions of Cohttp and Js_of_ocaml.

• Improve display of programs and logs in the WebUI.
• Add more User-level notifications for async-errors.
• Add indexes to the DB.
• Improve node-list display in the TextUI.
• Remove and clean-out some code.
• Add option ~safe_ids to job submission (on by default).
• Fix build with Lwt ≥ 3.0.0.
• Improve “Getting Started” documentation.

All the changes are mostly related to internal performance improvements, and removing code:

• Completely rewritten “persistence” backend: directly based on a PostgreSQL database, the postgresql library is still optional for building (client-only builds do not require libpq). This results in many performance improvements both in the UI and for the scalability of the engine (can handle workflows with 100k+ nodes).
• “Getting Started” documentation based on docker-compose.
• Killing orphan nodes happens now less “often,” this lets the engine focus on more interesting tasks.
• Improve UI filter queries (remove confusing function, allow more results per page).
• Remove the SSH tunneling system (SSH servers do not scale for that task, the feature never worked reliably).
• Remove the standalone mode of the engine; now Ketrew only works as a server.
• Remove daemonization code; please use your system's interface, or Docker.

Long awaited major release, we have been using the master branch for about a year with Biokepi, Epidisco, and lately Coclobas:

• The EDSL has been re-architectured around the notions of workflow_node and edges (the old EDSL.target API is still available but deprecated).
• Ketrew can build & run with ssl, tls, or no TLS implementation.
• Ketrew, thanks to Trakeva, can use Sqlite3 or PostgreSQL database backends.
• Lots of (Web)UI, plugin, performance improvements.

Main changes:

• Add WebUI.
• Make ketrew init much more clever.
• Allow Non-TLS HTTP.
• Improve YARN backend (incl. talking a bit its HTTP API).
• Do minor fixes and performance improvements.

This release fixes the build with OCaml ≥ 4.02.2 which introduced backards incompatible syntax changes for ppx-attributes.

Changes:

• Much more testing, with bisect_ppx coverage, and adding a 4-nodes virtual Hadoop cluster to the integration tests.
• 84b5ec0 Improve error message in generic_ssh_exec.
• 71b51b2 Cohttp 0.18.0 compatibility (PR #186 by @rgrinberg).
• 8f3b881 Put a bound on successive non-fatal failures per target.
• 8da0c94 Make target equivalence a symmetric relation.
• f599da6 Improve documentation of the EDSL (backported to the 1.0.0 website).
• 4d96eec Add new option to daemonize: ?no_log_is_ok (serialization is backwards compatible with 1.0.0).
• bec6de0 Free YARN backend from shared filesystems.

This version is very different form 0.0.0, it's faster & stronger and all. It breaks backwards compatibility at every level.

• Porting code using the EDSL should be straightforward following type-errors.
• The persistence backend was completely changed and cannot be used across versions.

This is the first release of Ketrew, a workflow engine, here are a few highlights:

• The EDSL can be used to build pretty complex workflows.
• Both client-server (HTTPS + JSON) and standalone modes are available.
• The implementation of the engine is naive/slow but testable & understandable.
• The command-line client can explore the state of workflows and pilot the engine interactively.
• Persistence of engine state is based on an “explorable” (but slow) git database.
• The engine talks through SSH to LSF and PBS clusters, and also to Unix machines that do not have a batch scheduler. More backends can be implemented through plugins to be linked with the library or using Dynlink (using direct paths or findlib names).
• See the documentation for more information.