Seed Job Support

Scale now has support for Seed-compliant jobs. Seed is a standard designed to encapsulate batch jobs within Docker containers. Seed allows for a Dockerized job to specify its inputs, outputs, resource requirements, etc. and enables easy and fast deployment with a Seed-compatible job framework (such as Scale). Legacy job types are now considered deprecated. After upgrading to Scale v5.6.0, users should transition all of their jobs to be Seed-compliant. In Scale v6.0.0, all support for legacy job types will be removed.

Dependencies

• DC/OS 1.10+
• Docker 1.13+
• ElasticSearch 2.4
• PostgreSQL 9.4+, PostGIS 2.0+
• Message Broker (RabbitMQ 3.6 or Amazon Web Services SQS)
• Vault 0.6.2+ (only if not using DC/OS Enterprise for secrets storage)

Breaking Changes

• N/A

Deprecated

• Legacy job types are now deprecated. Please transition all of your job types to be Seed-compliant.
• The v4 version of the REST API is now deprecated. We recommend that you migrate all use of the Scale REST API to use the new v5 REST API. The changes between the v4 and v5 versions are:
  1. The /v4/sources/{id}/ API has been replaced by a v5 version. The new v5 version removes the "ingests" and "products" fields, which were expensive to include. To get the same information, users should now use /v5/sources/{id}/ingests/, /v5/sources/{id}/jobs/, and /v5/sources/{id}/products/. The new /v5/sources/{id}/ API also removes the ability to use the file name in the URL.
  2. The /v4/status/ API has been replaced by a v5 version. The new v5 version is designed to be a near-real time look at the scheduler status and updates approximately every 5 seconds.
  3. The /v4/nodes/ API has been replaced with a v5 version. The new v5 version removes the port, slave_id, is_paused_errors, and last_offer fields in the response.
  4. The /v4/nodes/{id}/ API has been replaced with a v5 version. The new v5 version removes the port, slave_id, is_paused_errors, last_offer, resources, disconnected, and job_exes_running fields in the response.
  5. The /v4/nodes/status/ API has been deprecated in favor of the new /v5/status/ API.

Known Issues

• N/A

New Features

• Added the missing job-type V6 API endpoints (Issue 1529)
• Added is_paused and unmet_resources to the job type base serializer (Issue 1528)
• Added support to allow periods and other characters in data type tags (Issue 1518)

Enhancements/Updates

• N/A

Bug Fixes

• Added a retry decoration when certain job-type messages were throwing deadlock exceptions (Issue 1102)

Database Migrations

These are relatively quick database changes applied using Django migrations to alter tables, constraints, indexes, etc. The migrations are applied during the start up of the Scale scheduler.

• The ingest data_type field was replaced with a data_type_tags field
• The source.ScaleFile data_type field was replaced with a data_type_tags field

Database Updates

These are long running database changes related to actual data row updates that run in the background in a Scale system task. These updates are performed while Scale is running so they don't increase down time when deploying a new Scale version.

• N/A

This is a patch release to address a bug with outstanding mesos offers. Scale would take resource offers from mesos and would not decline unused offers, preventing other frameworks from using them.

Dependencies

• DC/OS 1.10+
• Docker 1.13+
• ElasticSearch 2.4
• PostgreSQL 9.4+, PostGIS 2.0+
• Message Broker (RabbitMQ 3.6 or Amazon Web Services SQS)
• Vault 0.6.2+ (only if not using DC/OS Enterprise for secrets storage)

Bug Fixes

• Fix for a bug where Scale would hold on to resource offers and keep other frameworks from using them. (Issue 1516)

Seed Job Support

Scale now has support for Seed-compliant jobs. Seed is a standard designed to encapsulate batch jobs within Docker containers. Seed allows for a Dockerized job to specify its inputs, outputs, resource requirements, etc. and enables easy and fast deployment with a Seed-compatible job framework (such as Scale). Legacy job types are now considered deprecated. After upgrading to Scale v5.6.0, users should transition all of their jobs to be Seed-compliant. In Scale v6.0.0, all support for legacy job types will be removed.

Dependencies

• DC/OS 1.10+
• Docker 1.13+
• ElasticSearch 2.4
• PostgreSQL 9.4+, PostGIS 2.0+
• Message Broker (RabbitMQ 3.6 or Amazon Web Services SQS)
• Vault 0.6.2+ (only if not using DC/OS Enterprise for secrets storage)

Breaking Changes

• The trigger rule system was removed for this release. Jobs and recipes will need to be scheduled via configuration from the workspace.

Future Deprecated (Do NOT include in release notes)

The v5 REST API has been deprecated and replaced by the v6 REST API. See the REST API documentation for all of the details on the v6 REST API. The following list details some of the major changes from v5 to v6:

• The /v5/jobs/executions/, /v5/job-executions/ and /v5/job-executions/{id}/ APIs are deprecated and have been replaced by /v6/jobs/{job-id}/executions/ and /v6/jobs/{job-id}/executions/{exe-num}/.
• The /v5/recipes/{id}/ API has been replaced by a v6 version. The new v6 version has a different response: the "inputs" section is removed, the "data" section is renamed to "input", and the "recipe_type" section will only contain "id", "name", "version", "title", and "description". The /v5/queue/new-recipe/ and /v5/recipes/{id}/reprocess/ APIs are also replaced with new v6 versions with the same changes in the response.
• The /v5/jobs/, /v5/jobs/{id}/, /v5/queue/new-job/, and /v5/queue/requeue-jobs/ APIs have been replaced with new v6 versions.
• The v5 sources, products, and import/export REST APIs have been deprecated. There are no v6 versions of these APIs.
• Deprecated the "reprocess_recipes" message in favor of a create-recipes message (Issue 1172)

Deprecated

• Legacy job types are now deprecated. Please transition all of your job types to be Seed-compliant.
• The v4 version of the REST API is now deprecated. We recommend that you migrate all use of the Scale REST API to use the new v5 REST API. The changes between the v4 and v5 versions are:
  1. The /v4/sources/{id}/ API has been replaced by a v5 version. The new v5 version removes the "ingests" and "products" fields, which were expensive to include. To get the same information, users should now use /v5/sources/{id}/ingests/, /v5/sources/{id}/jobs/, and /v5/sources/{id}/products/. The new /v5/sources/{id}/ API also removes the ability to use the file name in the URL.
  2. The /v4/status/ API has been replaced by a v5 version. The new v5 version is designed to be a near-real time look at the scheduler status and updates approximately every 5 seconds.
  3. The /v4/nodes/ API has been replaced with a v5 version. The new v5 version removes the port, slave_id, is_paused_errors, and last_offer fields in the response.
  4. The /v4/nodes/{id}/ API has been replaced with a v5 version. The new v5 version removes the port, slave_id, is_paused_errors, last_offer, resources, disconnected, and job_exes_running fields in the response.
  5. The /v4/nodes/status/ API has been deprecated in favor of the new /v5/status/ API.

Known Issues

• N/A

New Features

• Removed the trigger system (Issue 1181)
• Added a warning to not run jobs where resources won't be met until the warning has expired (Issue 1511)
• Moved the v6 job-types/ endpoint to job-type-names/ The new job-types/ endpoint now contains all versions of job types instead of a condensed version (Issue 1503)
• Implemented recipe if/else conditions (Issue 1488)
• Implemented a warning for non-standard resource names defined in the manifest (Issue 1493)
• Added a warning for jobs that will not be run due to resources that will never be fulfilled (Issue 1492)

Enhancements/Updates

• N/A

Bug Fixes

• Fixed a bug where requeueing jobs that were already queued left them in a limbo status (Issue 1494)

Database Migrations

These are relatively quick database changes applied using Django migrations to alter tables, constraints, indexes, etc. The migrations are applied during the start up of the Scale scheduler.

• N/A

Database Updates

These are long running database changes related to actual data row updates that run in the background in a Scale system task. These updates are performed while Scale is running so they don't increase down time when deploying a new Scale version.

• N/A

This is an optional patch release to mitigate RunC container escape vulnerability. If you have upgraded your underlying Docker Engine to 18.09.2, this is not a priority. Algorithm containers will now be launched as the nobody user. The next major release of Scale will enforce this behavior.

Dependencies

• DC/OS 1.10+
• Docker 1.13+
• ElasticSearch 2.4
• PostgreSQL 9.4+, PostGIS 2.0+
• Message Broker (RabbitMQ 3.6 or Amazon Web Services SQS)
• Vault 0.6.2+ (only if not using DC/OS Enterprise for secrets storage)

Bug Fixes

• Fix for RunC vulnerability allowing container escape from user algorithms. (Issue 1498)

This is an optional patch release to mitigate RunC container escape vulnerability. If you have upgraded your underlying Docker Engine to 18.09.2, this is not a priority. Algorithm containers will now be launched as the nobody user. The next major release of Scale will enforce this behavior.

Dependencies

• DC/OS 1.10+
• Docker 1.13+
• ElasticSearch 2.4
• PostgreSQL 9.4+, PostGIS 2.0+
• Message Broker (RabbitMQ 3.6 or Amazon Web Services SQS)
• Vault 0.6.2+ (only if not using DC/OS Enterprise for secrets storage)

Bug Fixes

• Fix for RunC vulnerability allowing container escape from user algorithms. (Issue 1498)

Seed Job Support

Scale now has support for Seed-compliant jobs. Seed is a standard designed to encapsulate batch jobs within Docker containers. Seed allows for a Dockerized job to specify its inputs, outputs, resource requirements, etc. and enables easy and fast deployment with a Seed-compatible job framework (such as Scale). Legacy job types are now considered deprecated. After upgrading to Scale v5.6.0, users should transition all of their jobs to be Seed-compliant. In Scale v6.0.0, all support for legacy job types will be removed.

Dependencies

• DC/OS 1.10+
• Docker 1.13+
• ElasticSearch 2.4
• PostgreSQL 9.4+, PostGIS 2.0+
• Message Broker (RabbitMQ 3.6 or Amazon Web Services SQS)
• Vault 0.6.2+ (only if not using DC/OS Enterprise for secrets storage)

Deprecated

• Legacy job types are now deprecated. Please transition all of your job types to be Seed-compliant.
• The v4 version of the REST API is now deprecated. We recommend that you migrate all use of the Scale REST API to use the new v5 REST API. The changes between the v4 and v5 versions are:
  1. The /v4/sources/{id}/ API has been replaced by a v5 version. The new v5 version removes the "ingests" and "products" fields, which were expensive to include. To get the same information, users should now use /v5/sources/{id}/ingests/, /v5/sources/{id}/jobs/, and /v5/sources/{id}/products/. The new /v5/sources/{id}/ API also removes the ability to use the file name in the URL.
  2. The /v4/status/ API has been replaced by a v5 version. The new v5 version is designed to be a near-real time look at the scheduler status and updates approximately every 5 seconds.
  3. The /v4/nodes/ API has been replaced with a v5 version. The new v5 version removes the port, slave_id, is_paused_errors, and last_offer fields in the response.
  4. The /v4/nodes/{id}/ API has been replaced with a v5 version. The new v5 version removes the port, slave_id, is_paused_errors, last_offer, resources, disconnected, and job_exes_running fields in the response.
  5. The /v4/nodes/status/ API has been deprecated in favor of the new /v5/status/ API.

Known Issues

• N/A

New Features

• Implemented transitional functionality to replace the trigger system (Issue 1485)
• Implemented a Scale system job for moving files (Issue 1161)

Enhancements/Updates

• N/A

Bug Fixes

• Fixed an issue where the V5 recipes details API would return a 500 error when confronted with a V6 recipe (Issue 1490)

Database Migrations

These are relatively quick database changes applied using Django migrations to alter tables, constraints, indexes, etc. The migrations are applied during the start up of the Scale scheduler.

• Added new table IngestEvent
• Added new field ingest_event to the job table
• job.event field is now nullable to prepare for trigger replacement
• Added new field ingest_event to the recipe table

Seed Job Support

Scale now has support for Seed-compliant jobs. Seed is a standard designed to encapsulate batch jobs within Docker containers. Seed allows for a Dockerized job to specify its inputs, outputs, resource requirements, etc. and enables easy and fast deployment with a Seed-compatible job framework (such as Scale). Legacy job types are now considered deprecated. After upgrading to Scale v5.6.0, users should transition all of their jobs to be Seed-compliant. In Scale v6.0.0, all support for legacy job types will be removed.

Dependencies

• DC/OS 1.10+
• Docker 1.13+
• ElasticSearch 2.4
• PostgreSQL 9.4+, PostGIS 2.0+
• Message Broker (RabbitMQ 3.6 or Amazon Web Services SQS)
• Vault 0.6.2+ (only if not using DC/OS Enterprise for secrets storage)

Breaking Changes

• The v4/ REST API endpoints were removed from this release.

Deprecated

• Legacy job types are now deprecated. Please transition all of your job types to be Seed-compliant.
• The v4 version of the REST API is now deprecated. We recommend that you migrate all use of the Scale REST API to use the new v5 REST API. The changes between the v4 and v5 versions are:
  1. The /v4/sources/{id}/ API has been replaced by a v5 version. The new v5 version removes the "ingests" and "products" fields, which were expensive to include. To get the same information, users should now use /v5/sources/{id}/ingests/, /v5/sources/{id}/jobs/, and /v5/sources/{id}/products/. The new /v5/sources/{id}/ API also removes the ability to use the file name in the URL.
  2. The /v4/status/ API has been replaced by a v5 version. The new v5 version is designed to be a near-real time look at the scheduler status and updates approximately every 5 seconds.
  3. The /v4/nodes/ API has been replaced with a v5 version. The new v5 version removes the port, slave_id, is_paused_errors, and last_offer fields in the response.
  4. The /v4/nodes/{id}/ API has been replaced with a v5 version. The new v5 version removes the port, slave_id, is_paused_errors, last_offer, resources, disconnected, and job_exes_running fields in the response.
  5. The /v4/nodes/status/ API has been deprecated in favor of the new /v5/status/ API.

Known Issues

• N/A

New Features

• Created a message to queue purge jobs for deleting a job's product files (Issue 1209)
• Created new v6 workspace REST APIs (Issue 1146)
• Implemented new command message to create recipes (handling reprocessing and sub-recipes as well) (Issue 1170)
• Implemented the setting of source meta-data fields on jobs (Issue 1275)
• Implemented a new command message to delete a source file for a purge (Issue 1211)
• Implemented a new command message for purging a recipe (Issue 1210)
• Implemented v6 REST API call to initiate a source file purge (Issue 1212)
• Implemented the creation of purge results to detail what was deleted during a source file purge (Issue 1126)
• Implemented a new command message that updates a recipe and handles sub-recipes (Issue 1171)
• Blocked v5 job type API editing of certain seed manifest properties (Issue 1334)
• Implemented v6 checking for Job Type's workspace in the pre-task (Issue 1321)
• Implemented the copying of job source meta-data fields to products (Issue 1194)
• Added additional source meta-data fields to the recipe model (Issue 1276]
• Added additional source meta-data fields to the job type API (Issue 1354)
• Added additional source meta-data fields to the v6 recipes REST API (Issue 1358)
• Implemented using the new v6 recipe logic (Issue 1177)
• Added the ability to stop a purge (Issue 1395)
• Implemented the new Recipe Condition model (Issue 1173)
• Added a new message for recipe conditions (Issue 1174)
• Added a new message to create recipe conditions (Issue 1382)
• Implemented handling the recipe conditions in an update_recipe message (Issue 1175)
• Updated the scheduler to use Mesos HTTP v1 Scheduler API (Issue 1040)
• Completed the v6 GET Recipe Types REST API (Issue 1379)
• Implemented the v6 PATCH/POST Recipe Types REST APIs (Issue 1178)
• Removed grep from docker commands (Issue 1448)
• Updated the status API to include node failed cleanup tasks (Issue 1444)
• Added ability to cancel jobs with name/version combo (Issue 1454)
• Added ability to requeue jobs with name/version combo (Issue 1434)
• Updated Amazon S3 Strike Version to list instead of single value (Issue 1450)
• Updated job-types REST API to filter job types by multiple keywords (Issue 1435)
• Updated recipe-types REST API to filter recipe types by multiple keywords (Issue 1440)
• Updated v6 job-types REST API to allow an ID query parameter (Issue 1390)
• Updated v6 job-types REST API to return a list of versions for the job type (Issue 1422)
• Added support for deployment within access controlled DCOS cluster (Issue 1439)
• Implemented the v6 queue REST API (Issue 1140)
• Added an additional environment variable to inject recipe metadata to a job runtime (Issue 1458)
• Implemented the v6 GET Recipe REST APIs (Issue 1179)
• Added tracking of the Vault status to the v5 and v6 status REST API (Issue 1359)
• Implemented recipe configuration (Issue 1106)
• Implemented the v6 reprocess recipe REST API (Issue 1381)
• Implemented data filters (Issue 1384)
• Updated v6 job-types validation to check for output-workspaces (Issue 1476)

Enhancements/Updates

• Changed UI filter table to default to a time window of 24 hours (Issue 1327)
• Added privileged label to the Job Types list (Issue 1362)

Bug Fixes

• Added a new column to allow correct version sorting for job types (Issue 1294)
• Fixed bug so that Seed jobs that do not provide a required output can result in the correct error being applied (Issue 1247)
• Fixed race condition bug where two jobs trying to create the same directory at the same time would result in an error (Issue 703)
• Fixed bug where Seed job types prevented the recipe editor from working (Issue 1350)
• Fixed bug where a node reset (during an offline or deprecated state) wasn't resetting all of the node state (Issue 1347)
• Fixed a bug where the purge process could delete additional source files (Issue 1342)
• Fixed a bug where the v5 Recipe-Type details api fails to convert the Seed job type manifest into a legacy job type interface by accessing a missing field. (Issue 1374)
• Fixed a bug where the Filter PurgeResults record edits on trigger_event and not source_file (Issue 1397)
• Fixed a bug where purge_jobs were modified to have the file_ancestry link deleted before job_exe (Issue 1394)
• Fixed a bug where the v6 Job Type REST API was returning a 500 Server Error for v5 Job Types (Issue 1375)
• Fixed a bug where the v5 Recipe-Type API returns a 500 error if 'inputs' or 'outputs' were not defined in the manifest (Issue 1400)
• Fixed a bug where expanding the node reset method resulted in the health check being None when performing checks on that variable later (Issue 1406)
• Fixed a bug where seed jobs with inputs defined to be multiple, but only receiving one input were incorrectly setting the injected input environment variable (Issue 1409 Issue 1430)
• Fixed a bug to correct the scale-delete-files fixture (Issue 1425)
• Fixed a bug where cleanup errors were returning 0 even when an error occurred (Issue 1443)
• Fixed a bug where the JobConfiguration was failing to validate against non-seed job types (Issue 1431)
• Fixed a bug where the recipe type details serializer was not returning the proper recipe type definition (Issue 1472)

Database Migrations

These are relatively quick database changes applied using Django migrations to alter tables, constraints, indexes, etc. The migrations are applied during the start up of the Scale scheduler.

• Adds new column version_array to table job_type
• Adds new columns source_collection, source_sensor, source_sensor_class, source_task, source_ended, and source_started to table job
• New table purge_results is created
• In the workspace table, the archived column is renamed to deprecated and the total_size and used_size columns are removed.

Seed Job Support

Scale now has support for Seed-compliant jobs. Seed is a standard designed to encapsulate batch jobs within Docker containers. Seed allows for a Dockerized job to specify its inputs, outputs, resource requirements, etc. and enables easy and fast deployment with a Seed-compatible job framework (such as Scale). Legacy job types are now considered deprecated. After upgrading to Scale v5.6.0, users should transition all of their jobs to be Seed-compliant. In Scale v6.0.0, all support for legacy job types will be removed.

Dependencies

• DC/OS 1.9+
• Docker 1.11+
• ElasticSearch 2.4
• PostgreSQL 9.4+, PostGIS 2.0+
• Message Broker (RabbitMQ 3.6 or Amazon Web Services SQS)
• Vault 0.6.2+ (only if not using DC/OS Enterprise for secrets storage)

Deprecated

• Legacy job types are now deprecated. Please transition all of your job types to be Seed-compliant.
• The v4 version of the REST API is now deprecated. We recommend that you migrate all use of the Scale REST API to use the new v5 REST API. The changes between the v4 and v5 versions are:
  1. The /v4/sources/{id}/ API has been replaced by a v5 version. The new v5 version removes the "ingests" and "products" fields, which were expensive to include. To get the same information, users should now use /v5/sources/{id}/ingests/, /v5/sources/{id}/jobs/, and /v5/sources/{id}/products/. The new /v5/sources/{id}/ API also removes the ability to use the file name in the URL.
  2. The /v4/status/ API has been replaced by a v5 version. The new v5 version is designed to be a near-real time look at the scheduler status and updates approximately every 5 seconds.
  3. The /v4/nodes/ API has been replaced with a v5 version. The new v5 version removes the port, slave_id, is_paused_errors, and last_offer fields in the response.
  4. The /v4/nodes/{id}/ API has been replaced with a v5 version. The new v5 version removes the port, slave_id, is_paused_errors, last_offer, resources, disconnected, and job_exes_running fields in the response.
  5. The /v4/nodes/status/ API has been deprecated in favor of the new /v5/status/ API.

Bug Fixes

• Fixed bug that broke ingest detail UI page (Issue 1326)

Seed Job Support

Scale now has support for Seed-compliant jobs. Seed is a standard designed to encapsulate batch jobs within Docker containers. Seed allows for a Dockerized job to specify its inputs, outputs, resource requirements, etc. and enables easy and fast deployment with a Seed-compatible job framework (such as Scale). Legacy job types are now considered deprecated. After upgrading to Scale v5.6.0, users should transition all of their jobs to be Seed-compliant. In Scale v6.0.0, all support for legacy job types will be removed.

Dependencies

• DC/OS 1.9+
• Docker 1.11+
• ElasticSearch 2.4
• PostgreSQL 9.4+, PostGIS 2.0+
• Message Broker (RabbitMQ 3.6 or Amazon Web Services SQS)
• Vault 0.6.2+ (only if not using DC/OS Enterprise for secrets storage)

Deprecated

• Legacy job types are now deprecated. Please transition all of your job types to be Seed-compliant.
• The v4 version of the REST API is now deprecated. We recommend that you migrate all use of the Scale REST API to use the new v5 REST API. The changes between the v4 and v5 versions are:
  1. The /v4/sources/{id}/ API has been replaced by a v5 version. The new v5 version removes the "ingests" and "products" fields, which were expensive to include. To get the same information, users should now use /v5/sources/{id}/ingests/, /v5/sources/{id}/jobs/, and /v5/sources/{id}/products/. The new /v5/sources/{id}/ API also removes the ability to use the file name in the URL.
  2. The /v4/status/ API has been replaced by a v5 version. The new v5 version is designed to be a near-real time look at the scheduler status and updates approximately every 5 seconds.
  3. The /v4/nodes/ API has been replaced with a v5 version. The new v5 version removes the port, slave_id, is_paused_errors, and last_offer fields in the response.
  4. The /v4/nodes/{id}/ API has been replaced with a v5 version. The new v5 version removes the port, slave_id, is_paused_errors, last_offer, resources, disconnected, and job_exes_running fields in the response.
  5. The /v4/nodes/status/ API has been deprecated in favor of the new /v5/status/ API.

New Features

• Created a new v6 job detail REST API (Issue 1116)
• Created a new v6 jobs list REST API (Issue 1113)
• Added a new background message that can purge jobs from the database (Issue 1207)
• Created new v6 job execution REST APIs (Issue 1221)
• Enabled Scale to operate with Mesos GPU resources (Issue 1051)

Enhancements/Updates

• The jobs UI page now allows separate filtering on job type name and version (Issue 1187, Issue 1300)
• Implemented the parsing and handling of Seed meta-data files for job output products (Issue 1160)
• Added source meta-data filtering to v6 files REST API (Issue 1274)
• Updated Scale UI overview page to use v5 status REST API (Issue 905, Issue 1283)
• Removed resource_level from v6 scheduler REST API (Issue 1284)
• Update to better handle Docker image names for each job type revision (Issue 1310)
• Added job type resource editing to Scale job type UI page (Issue 1206)

Bug Fixes

• Fixed a bug in the v6 job types REST API (Issue 1291)
• Fixed a bug in the POST job type REST API that prevented a legacy job type from being updated to a Seed-compliant job type (Issue 1285)
• Fixed a bug with detecting Seed meta-data JSON as a product file (Issue 1316)
• Fixed a bug with submitting a Seed job type with no defined tags (Issue 1307)
• Fixed bugs related to handling the Seed manifest (Issue 1301)

Database Migrations

These are relatively quick database changes applied using Django migrations to alter tables, constraints, indexes, etc. The migrations are applied during the start up of the Scale scheduler.

• New fields source_collection, source_sensor, source_sensor_class, and source_task are added to the scale_file table, along with indexes on those columns. The time to complete this will depend upon the size of your scale_file table.
• Removed field resource_level from the scheduler table
• Added new field docker_image to the job_exe table
• Added new field docker_image to the job_type_revision table
• Added new field docker_image to the queue table

Database Updates

These are long running database changes related to actual data row updates that run in the background in a Scale system task. These updates are performed while Scale is running so they don't increase down time when deploying a new Scale version.

• N/A

Seed Job Support

Starting with this release, Scale now has support for Seed-compliant jobs. Seed is a standard designed to encapsulate batch jobs within Docker containers. Seed allows for a Dockerized job to specify its inputs, outputs, resource requirements, etc. and enables easy and fast deployment with a Seed-compatible job framework (such as Scale). Legacy job types are now considered deprecated. After upgrading to Scale v5.6.0, users should transition all of their jobs to be Seed-compliant. In Scale v6.0.0, all support for legacy job types will be removed.

Dependencies

• DC/OS 1.9+
• Docker 1.11+
• ElasticSearch 2.4
• PostgreSQL 9.4+, PostGIS 2.0+
• Message Broker (RabbitMQ 3.6 or Amazon Web Services SQS)
• Vault 0.6.2+ (only if not using DC/OS Enterprise for secrets storage)

Deprecated

• Legacy job types are now deprecated. Please transition all of your job types to be Seed-compliant.
• The v4 version of the REST API is now deprecated. We recommend that you migrate all use of the Scale REST API to use the new v5 REST API. The changes between the v4 and v5 versions are:
  1. The /v4/sources/{id}/ API has been replaced by a v5 version. The new v5 version removes the "ingests" and "products" fields, which were expensive to include. To get the same information, users should now use /v5/sources/{id}/ingests/, /v5/sources/{id}/jobs/, and /v5/sources/{id}/products/. The new /v5/sources/{id}/ API also removes the ability to use the file name in the URL.
  2. The /v4/status/ API has been replaced by a v5 version. The new v5 version is designed to be a near-real time look at the scheduler status and updates approximately every 5 seconds.
  3. The /v4/nodes/ API has been replaced with a v5 version. The new v5 version removes the port, slave_id, is_paused_errors, and last_offer fields in the response.
  4. The /v4/nodes/{id}/ API has been replaced with a v5 version. The new v5 version removes the port, slave_id, is_paused_errors, last_offer, resources, disconnected, and job_exes_running fields in the response.
  5. The /v4/nodes/status/ API has been deprecated in favor of the new /v5/status/ API.

New Features

• Created new v6 version of the file REST APIs (Issue 1141)
• Created new v6 version of the Strike REST APIs (Issue 1143)
• Created new v6 version of the Scan REST APIs (Issue 1142)
• Created new v6 version of the Job Type REST APIs (Issue 1218, Issue 1219)
• Created new v6 version of the Error REST APIs (Issue 1134)
• Scale now has the capability to run Seed-compliant jobs. See https://ngageoint.github.io/seed/ for more information on the Seed specification. (Issue 971, Issue 1225, Issue 1232, Issue 1234, Issue 1236, Issue 1238, Issue 1240, Issue 1245)
• Created new message to handle creating jobs (Issue 1169)

Enhancements/Updates

• Improved recipe metrics to handle sub-recipes (Issue 1165)
• Improved batch metrics to handle sub-recipes (Issue 1166)
• Updated message for processing job input to handle sub-recipes (Issue 1167)
• Updated message for processing recipe input to handle sub-recipes (Issue 1168)

Bug Fixes

• Fixed a bug where the bulk requeue did not correctly handle canceled jobs that had never been queued before (Issue 1254)
• Fixed bugs relating to how Seed command variable expansion worked (Issue 1248, Issue 1252)
• Fixed bug related to the v5 job details REST API (Issue 1242)
• Fixed vulnerability in the cryptography dependency (Issue 1257)

Database Migrations

These are relatively quick database changes applied using Django migrations to alter tables, constraints, indexes, etc. The migrations are applied during the start up of the Scale scheduler.

• New fields recipe_id, root_recipe_id, sub_recipes_total, and sub_recipes_completed are added to the recipe table.
• The error table is updated to handle Seed errors: new field job_type_name is added and the unique index on name is changed to a unique multi-column index on (job_type_name, name).
• The following job_type fields are deprecated and are now nullable: cpus_required, custom_resources, disk_out_const_required, disk_out_mult_required, docker_params, docker_privileged, error_mapping, mem_const_required, mem_mult_required, shared_mem_required, timeout, and uses_docker.
• The job_type field archived is renamed to deprecated and interface is renamed to manifest. Also in the job_type_revision table interface is renamed to manifest.
• A new table named job_type_tag is created.
• The following job fields are deprecated and are now nullable: priority, and timeout.
• In the recipe_input_file table the scale_file_id field is renamed to input_file_id.