WALKOFF Versions Save

Added

• CSV to Array action in the Utilities app

Changed

• The action results SSE stream truncates the result using the MAX_STREAM_RESULTS_SIZE_KB config option

Fixed

• Bytes conversion bug in the RedisCacheAdapter
• Bug in playbook editor using users and roles as arguments
• Bug where some callbacks weren't getting registered
• Column width bug in playbook editor, execution, and metrics pages
• OpenAPI validation bug with the newest version of the swagger validator

Added

• Arguments can now reference branches. This will resolve to the number of times that branch has been executed.
• Log messages are more comprehensive and useful.
• More error checking on the worker processes to harden them.

Fixed

• Bug where databases couldn't be used with a password.
• Bug where app instances would receive an Argument rather than the necessary integer ID.
• Compatibility issue with pip 10 and the install_dependencies.py script.
• Bug in the validation of execution elements where, once an error was found, it wouldn't be removed.
• Fixed bug where exporting playbooks with Python 3 would cause an error.
• Bug where argument ids were not stripped on exporting of playbooks, causing errors when importing them into a different instance of Walkoff.

Fixed

• Bug where Workflows with unbounded Actions were unable to be executed

Added

• Multiple tools have been added to help develop workflows
  • Playbooks can be saved even if they are invalid. However, playbooks cannot be executed if they are invalid.
  • The playbook editor displays the errors on a workflow which must be solved before the workflow can be executed
  • You can now use Python's builtin logging module in an app, and the log messages will be displayed in the playbook editor
• The metrics page has been introduced in the UI which displays simple metrics related to the execution of workflows and actions.
• The devices used in the actions in workflows are now objects, enabling dynamic selection of the device used for the action. To further support this, an action in the Utilities app named get devices by fields allows you to query the devices database.
• The ability to use a key-value storage has been created. This is now the mechanism used to push workflows and backs the SSE streams. Currently two options are available for key-value store, DiskCache, a SQLite-backed key-value storage, and Redis. By default Walkoff will use DiskCache, but it is recommended that users configure and use Redis.
• The SSEs now use dedicated SseStream objects which are backed by the cache. These objects make constructing and using streams much easier. walkoff.see.InterfaceSseStream and walkoff.sse.FilteredInterfaceSseStream objects have been made available to use in custom interfaces.
• A CaseLogger object which makes it much easier to log events to the case database has been created.

Changed

• The interfaces.AppBlueprint used to construct interfaces has been modified to extend from walkoff.sse.StreamableBlueprint which in turn extends Flask's Blueprint. This makes the interface cleaner and more flexible.
• Changes to the REST API
  • In the configuration resource:
    • workflow_path, logging_config_file, and zmq_requests have been removed from the API
    • The ability to edit the cache configuration has been added
  • In the playbook resources:
    • All execution elements have a read only list of human-readable errors
    • A workflow has a read only Boolean field "is_valid" which indicates if any of its execution elements have errors
• All changes to the configuration will only be applied on server restart
• Refactorings have been done to minimize the amount of global state used throughout Walkoff. Work will continue on this effort.
• Metrics are now stored in the execution database
• Changes to styling on the playbook editor

Deprecated

• walkoff.helpers.create_sse_event has been deprecated and will be removed in version 0.10.0. Use walkoff.sse.SseEvent or the streams in walkoff.sse instead .

Fixed

• Bug where branches where all branches weren't being evaluated in a workflow
• Bug where object arguments could not be converted from strings

Contributor

• Testing the backend now requires the additional the dependencies in requirements-test.txt
• The minimum accepted unit test coverage for the Python backend is now 88%

Fixed

• Bug where some device fields were being deleted on update

Fixed

• Bug where NO_CONTENT return codes were failing on Werkzeug WSGI 0.14

Changed

• All node modules are now bundled into webpack

Fixed

• An unintentional backward-breaking change was made to the format of the dictionary used in the interface dispatcher which sometimes resulted in a dict with a "data" field inside a "data" field. This has been fixed.

Changed

• Improved deserialization in the user interface
• Empty arrays are omitted from returned execution element JSON structure in the REST API.

Fixed

• PATCH /api/devices now doesn't validate that all the fields of the device are provided.
• Fixed dependency bug on GoogleProtocolBuffer version

Introducing Conditional Expressions for complete control over conditions, Execution Control to monitor and control running workflows, performance improvements, and an improved REST API.

Added

• An execution control page is now available on the user interface. This page allows you to start, pause, resume, and abort workflows as well as displays the status of all running and pending workflows.
  • With this feature is a new resource named workflowqueue which is available through the /api/workflowqueue endpoints.
• You now have the ability to use a full set of Boolean logic on conditions. This means that on branches and triggers you can specify a list of conditions which must all be true (AND operator), or a list of conditions of which any must be true (OR operator), or a list of conditions of which exactly one must be true (XOR operator). You can also negate conditions or have child conditions. This new conditional structure is called a ConditionalExpression and wraps the old Condition objects.
• Playbooks can be exported to and imported from a JSON text file through either the UI or using the new GET /api/playbooks?mode=export and the POST /api/playbooks using a multipart/form-data body respectively.

Changed

• Significant changes to the REST API
  • We have changed the HTTP verbs used for the REST API to reflect their more widely-accepted RESTful usage. Specifically, the POST and PUT verbs have been swapped for most of the endpoints.
  • Workflows are now accessed through the new /api/workflows endpoints rather than the /api/playbooks endpoints
  • The /api/playbooks and the /api/workflows endpoints now use the UUID instead of the name.
  • The /api/playbook/{id}/copy and the /api/playbooks/{id}/workflows/{id}/copy endpoints are now accessed through POST /api/playbooks?source={id_to_copy} and the POST /api/workflows?source={id_to_copy} endpoints respectively.
  • Server-Sent Event streams are now located in the /api/streams endpoints
  • Errors are now returned using the RFC 7807 Problem Details standard
• Playbooks, workflows, and their associated execution elements are now stored in the database which formerly only held the devices. The both greatly increased scalability as well as simplified the interactions between the server and the worker processes as well as increased scalability.
• Paused workflows and workflows awaiting trigger data are now pickled (serialized to binary) and stored in a database table. Before, a conditional wait was used to pause the execution of a workflow. By storing the state to the database, all threads on all worker processes are free to execute workflows.
• Information about the workflow which sent events are now available in both the Google Protocol Buffer messages as well as the arguments to callbacks using the interface event dispatcher.
• All times are stored in UTC time and represented in RFC 3339 format
• The marshmallow object serialization library is now used to serialize and deserialize execution elements instead of our old homemade solution

Deprecated

• The "sender_uids" argument in the interface dispatcher on_xyz_event decorators is now an alias for "sender_ids". This will be removed in version 0.9.0

Removed

• The /api/playbooks/{name}/workflows/{name}/save endpoint has been removed.
• The /api/playbooks/{name}/workflows/{name}/{execute/pause/resume} endpoints have been removed. Use the /api/workflowqueue resource instead
• Removed workflow_version from the playbooks. This may be added later to provide backwards-compatible import functionality to the workflows.
• /api/devices/import and /api/devices/export endpoints have been removed. Use the new POST /api/devices with multipart/form-data and GET /api/devices?mode=export endpoints respectively.

Contributor

• The minimum accepted unit test coverage for the Python backend is now 86%

Fixed

• Fixed bug in create_sse_event where data field of the SSE would not be populated if no data was not specified, causing the SSE event to be invalid