WALKOFF Versions Save

[1.0.0-rc.2]

This update includes a rewrite of the API Gateway from Flask to FastAPI to leverage Python Asyncio functionality. Server-Sent Events used for streaming data to the UI was also replaced by Socket.IO which should be easier to maintain and be more robust. UI improvements for interacting with JSON data were also included.

We would have liked to go for a full 1.0.0 release within the month, but due to factors outside of our control, the development team will be pausing work on WALKOFF. We hope to be back within the next couple weeks.

Added

• Docker logs are now streamed to the UI when rebuilding apps for better feedback on errors.
• JSON editor/viewer throughout UI where JSON can be accepted (e.g. local/global variables, action results, etc.).
• RBAC for App Editor.
• Upload file capability in App Editor.
• Moved certain items in navigation menu to sub-menu.
• End-to-end testing suite for API.
• Travis CI testing reimplemented (Appveyor in progress).

Changed

• Flask framework replaced with FastAPI async framework.
• Using Pydantic models for (de)serialization and validation to/from MongoDB.
• Server-Sent Events replaced with Socket.IO.
• PostgreSQL database replaced with MongoDB.
• Internal HTTP communication removed in favor of Redis and Socket.IO.
• All passwords for resources now randomly generated upon first run/build and stored in Docker Secrets.

Fixed

• UI should no longer intermittently fail to read from database (including authentication issues).
• Workflow status now correctly reports current app/action executing.

[1.0.0-rc.1]

This update includes a number of bugfixes, particularly to role-based permissions and the workflow scheduler.

Added

• Minio data from the WALKOFF application editor is now persistent after bringing WALKOFF down.

• Personal user settings have been added to the drop-down menu under a user profile. This new feature allows for all users to change their own username and password.

• mitre_attack and adversary_hunting applications have been added to WALKOFF’s default applications. These two apps have pre-configured actions that run Powershell scripts that may be useful to blue teams/threat hunting.

• Added subcommand to bootloader for rebuilding and updating specific services.

• Added a three-tier level role permissions system for Global Variable and Workflow creation. A user can choose between “only-me,” “everyone,” and “role-based” options for their access restriction level.

• Added a close button to Action Settings menu during workflow creation.

• Added sample_report_data action to basics application. This app generates sample CSV data to be imported into the Reports tab.

• Added Unsaved Changes warning on Workflow and Application editor pages.

• Added ability to create/edit Global Variables from the workflow editor.

Changed

• Top bar navigation has been reworked to include a “Settings” drop-down menu. Global variable creation, user settings, workflow scheduler creation, and link to Portainer have been moved to this menu.

• Changed Workflow Variables to Local Variables and always display them in the workflow editor.

• Renamed hello_world application tobasics

Security

• File access in apps is now handled with a context manager.

Fixed

• Workflow abortion in the execution tab is now functional and properly ends the workflow stream.

• The unpacking zipped apps on startup was reimplemented with bootloader changes.

• The execution tab now returns started_at time for each NodeStatusMessage for a WorkflowStatusMessage. Likewise, current_app and current_action are now live updated during Workflow processing.

• Workflow scheduler’s tasks can now be updated with new variables, descriptions, and titles. Interval scheduled tasks no longer become stuck in PENDING.

• Fixed the ability undo/redo the deletion of an action on the workflow editor.

Removed

• Individual tasks can no longer be toggled between pause and start in the scheduler.

[1.0.0-beta.1]

This update introduces a number of new features, including an App Editor in the UI, more granular role-based permissions, and a "bootloader" for automating deployment of WALKOFF.

Included are the massive re-architecture changes from 1.0.0-alpha.1 and 1.0.0-alpha.2 for which we did not make a GitHub release (changelogs below).

Added

• App Editor for editing app files and building Docker images from said apps. You can use this to change apps while WALKOFF is running, without restarting the whole stack.
• More granular RBAC where permissions on individual workflows and global variables can be restricted to specific roles.
• Execution results display on individual nodes to aid in identifying results.
• Bootloader container to automate deployment and teardown of the WALKOFF stack.
• Autogenerated walkoff_client Python package for interfacing with WALKOFF API. Work in progress.
• JSON editor GUI for editing arrays and objects in action parameters.
• JSON editor GUI for editing Workflow and Global Variables

Changed

• Dashboards renamed to Reports. Lots of work still to do here.
• Condition and Transform exceptions now get passed up into workflow results for easier debugging.
• Common config location to minimize number of locations where ports, service names, etc. need to change when configuring WALKOFF.
• WALKOFF now runs only on Docker Swarm, no longer with plain Docker Compose.
• All services now follow a walkoff_core, walkoff_app, or walkoff_resource naming scheme to disambiguate services from other stacks.
• Exceptions in Apps are now propagated correctly to action results.
• All WALKOFF services that need to be exposed now route through NGINX.

Security

• WALKOFF now uses HTTPS behind NGINX using a self-signed certificate.

Fixed

• Worker's workflow_types are now tagged with a _walkoff_type to prevent ambiguity with user provided data
• SSH app updated to allow more conventional use of wildcards, relative/absolute paths, etc.
• Conditions no longer cause entire subtrees to be cancelled, only the immediate successors.
• walkoff_default Docker network is attachable by default for external services to attach to us.
• Endpoints for PUT now correctly use resource IDs in path parameters.
• Importing a workflow with the same name as an existing one should no longer overwrite the original.
• Umpire scaling heartbeat slowed down to reduce race conditions - will be replaced with on-demand, resource aware scaling in future.
• Queued actions are correctly cleaned up when aborting a workflow.
• Enforced startup order of all the services to avoid busywaiting when services aren't up.
• Reduced intensity of "Server not responding" pop-up.
• Database commit issues relating to Workflow errors resolved.
• Parameters are correctly passed through to node status messages on the frontend.
• Actions now enforce results being JSON serializable, preventing issues with serializing Python objects.
• Action console logger reconnected to frontend.

[1.0.0-alpha.2]

This update includes numerous bugfixes and a number of reintroduced features. This release wasn't tagged, so the changelog is included here.

Added

• Trigger nodes allow you to pause workflow execution until webhook for the trigger is hit with data
• Basic Condition nodes allow you to perform branching execution in a more flowchart-like manner
• Basic Transform nodes allow you to write code snippets to transform/remap/select action results on the fly (UI support pending)
• Parallel Action node types in the workflow editor allow you to parallelize actions on a specified parameter
• Display UUIDs for workflow and workflow nodes in UI
• Portainer container creates UI for docker management

Changed

• WALKOFF now runs utilizing stack deploy, allowing for the use of external Docker secrets
• App version no longer required in app_name in api.tyaml
• CRUD endpoints now accept resource names as keys when applicable
• Globals can now be arbitrary JSON (UI support pending)
• Builtins build location moved to Umpire and is only built once on startup

Removed

Security

• Implemented AES-256 encryption/decryption for Global Variables. Exclusive-access decryption based on account level standing still needs to be implemented in the future. Currently, any GET request to the API gateway will return a decrypted Global Variable, regardless of account.

Fixed

• Workflow import/export
• Workflow validation (still needs work); workflows can be saved in an incomplete state again
• Validate workfow name uniqueness when creating workflows
• Testing suite (still needs expansion)
• Uniqueness constraints on CRUD operations
• Dereferencing Global and Workflow variables in workflows
• Ability to override starting parameters in a workflow execution
• Ability to update/delete encrypted Global Variables
• Hide global values by default on Globals tab
• Default boolean parameters to false
• Copying and pasting of nodes in workflow editor
• Accessing action results before conditionals in parameters that follow it

[1.0.0-alpha.1]

This update includes a near-complete rewrite of the workflow execution logic, and a considerable refactor of the server in preparation for a future move to an asynchronous framework. The following changes are not exhaustive.

Added

• "Umpire" added, which handles building and replication of Worker and App containers.

Changed

• Docker Compose is required. Python 3.7 is required if running components locally (primarily for development).
• Execution logic completely rewritten to support containerized architecture from the ground up.
• Apps now live in their own containers, separate from workers.
• Apps should now be (internally) stateless
• Kubernetes support has been removed in favor of using Docker Swarm API.
• Playbooks removed, Workflows can now be grouped by tags instead.
• unittest has been replaced with pytest
• Redis is now the primary communication channel between components (removing ZMQ and Kafka).
• SQLite database should no longer be used if running locally for development.
• Workflow Execution page has been overhauled aesthetically.

Removed

• Triggers have been temporarily removed, but are targeted for a near-future 1.1 release.

[0.9.4]

2018-12-11

Added

• Added ability to view WALKOFF Server API locally via /api/docs with the server running.

Fixed

• Execution DB now gets properly closed when WALKOFF exits. Fixes issues with docker-compose stop/start.
• Triggers on unbound actions (apps without devices) fixed.
• Add Docker image and compose file based on development branch.
• Upgraded WALKOFF Server API from swagger2 to openapi3, which includes improved security, and better request validation.
• Upgraded Python marshmallow library version, which includes stricter validation.
• Please note: because some dependency library versions were changed in the requirements.txt file, users must run the command pip install --upgrade -r requirements.txt to make sure all dependencies are met. This is also good practice to do after every new release.

[0.9.3]

2018-12-03

This is a minor release to fix missing front-end resources. A number of documentation changes have also been made, particularly regarding installing WALKOFF on Windows, as running WALKOFF directly on Windows has no longer supported since 0.9.0.

Fixed

• References to running WALKOFF directly on Windows now emphasize lack of support.
• Front-end dependencies have been added to the repository.

[0.9.2]

2018-11-30

This is a minor release primarily to ease installation of WALKOFF.

Added

• README.md contains further documentation on running WALKOFF locally, in Docker, or in Kubernetes
• NodeJS and NPM are no longer required, as the front-end components are now prepackaged in the main repository.

[0.9.1]

2018-11-26

Added

• README.md now contains more detailed instructions on using WALKOFF with Docker, as well as a docker-compose file
• All databases will now be stamped with the most up-to-date alembic version, so WALKOFF will not run if you are using an out-of-date database (see Fixed section for more details)

Fixed

• When using Redis as an external accumulator, results are now pickled to preserve typing. This fixes the issue where everything (list, int, etc.) was incorrectly being returned as strings
• Fixed walkoffctl update script to correctly update databases -- run python -m walkoff local update to update
• ActionResult objects are now pretty-printed correctly in the console and log files
• Python Redis library is now pinned in requirements.txt due to breaking changes
• Fixed certificate generation for Kubernetes certificates

Removed

• Update.py script was removed and replaced with walkoffctl update (see Fixed section for more details)

[0.9.0]

2018-11-14

Please Note: From version 0.9.0 forwards, WALKOFF requires a Redis cache to operate. You can run Redis natively on most Linux distributions (see the Redis quickstart guide: https://redis.io/topics/quickstart or search for a packagein your OS's package manager). On Windows, you will need to use Docker to run Redis in a container or expose Redis from a VM.

Added

• Support for running WALKOFF in a Kubernetes cluster using docker images and helm
• Command line interface (walkoffctl) for managing WALKOFF installations, both locally and on Kubernetes
• More comprehensive logging with Prometheus, fluentd, and flask
• Ability to run apps in separate containers for better scalability (still in-progress)
  • Support for templating Docker files for apps and runtimes
• Began introducing PyTest as a more maintainable alternative to unittest
• Support for tracking which User executed which Workflow
• Now have the option of choosing ZMQ sockets or Kafka message queues to communicate with executing Workflows

Changed

• Moved logic of executing a Workflow into Workflow execution context objects, which allows for more flexiblity
• Upgraded to boostrap4 and Angular 6.1.7

Removed

• Removed support for DiskCache -- a Redis cache must be installed to run WALKOFF
• The Case database, as it was unnecessary on top of the comprehensive logging done by WALKOFF

Fixed

• Fixed the foreign key constraints in both databases (they were not being enforced previously)
• Connexion library version updated in requirements.txt to fix access_token and 404 error
• Minor aesthetic improvements to front-end

Fixed

• Fixed a bug caused by a new version of the connexion library which made the OpenAPI specification invalid

Added

• Workflows now support environment variables. These are top-level arguments to a workflow. These are then exposed on the execution page allowing users to modify the most important variables in a workflow without modifying the workflow itself.
• Added a health check endpoint at the /heath endpoint

Changed

• The Metrics page now defaults to showing workflow metrics instead of app metrics

Fixed

• Action results and workflow results stream now filter for the currently-executing workflow. This eliminates many issues experienced by multiple users executing workflows concurrently from the workflow editor
• Fixed an error which caused the Scheduler to not execute workflows
• Fixed another bug in the scheduler in which the scheduled workflows would not persist across server restarts
• A bug where messages couldn't be sent
• A bug where modifying more than one device at a time on the playbook editor would cause the workflow to be invalidated
• Some database configuration bugs when used with non-SQLite databases
• Fixed a bug which wouldn't allow a user to a abort a workflow if it was pending execution.