Concourse Versions Save

Concourse is a container-based continuous thing-doer written in Go.

v7.8.1

1 year ago

🐞 Bug Fixes

  • Fixed a bug of error "invalidated-worker-resource-cache-exists" in get step (#8417) @evanchaoli :link:

📦 Bundled resource types

v7.8.0

1 year ago

🚨 Breaking

  • Do not cache secrets indefinitely when using Vault KV v2 (#7955) @sprsquish :link:
    For who uses Vault KV v2 as their creds manager, this change eliminates the ability to set an infinite cache duration, which may be a bug others are relying on.

✈️ Features

  • Add --team to fly watch (#8171) @dhantha :link:

    • Added team flag to fly watch command, you could use this in the following format fly -t example watch --job my-pipeline/tests --build 52 --team other-team

  • Add no-input-strategy and let get/check use it. (#8183) @evanchaoli :link:

    • Add optional flag --no-input-container-placement-strategy for configuring a container placement strategy used for only get and nested check steps. Configuring this strategy will prevent get/check step to be placed to a busy worker.

  • Optimize worker selection when global-resources is enabled (#8184) @evanchaoli :link:

  • Add fly clear-versions command (#8196) @clarafu :link:

    • Can be used to clear version history of a resource or resource type
    • Can only be used by an admin user
    • If you have global-resources enabled, it can possibly delete version histories of other resources/resource-types in other pipelines so there is a warning message that will show any resources or resource types that are affected.

  • Enhance locks (#8198) @evanchaoli :link:

    • Optimize work load distribution across ATCs.

  • Add prometheus emitter for jobs scheduled duration (#8276) @maxknee :link:

  • Expose ATC_EXTERNAL_URL to task env. (#8277) @evanchaoli :link:

  • Avoid duplicating parallel volume streams (#8322) @andy-paine from Depop :link:

    • Steps that stream volumes will now use a global (per worker) lock to ensure identical volumes are not streamed more times than they need to be
    • A new waiting-for-streamed-volume/waiting for volume <name> to be streamed by another step event is included in build step logs where this behavior occurs

  • Force checks on nested resource types when manually triggered build (#8342) @clarafu :link:

    • When a build is manually triggered, it will cause any nested resource types or images to skip its checking interval, essentially forcing a check. This will not result in the same resource type getting checked multiple times if it appears multiple times in a build.

🐞 Bug Fixes

  • Reset resource config IDs on save (#8002) @rjinskidepop :link:

    • Fixes #7468 where wrong resource config was used

  • Show var source error on resource and build page (#8015) @xtremerui :link:

    • Now error caused by variable interpolation can be shown correctly on resource and build page.

  • Fix a bug when a worker is pruned, volumes streamed from it got destroyed immediately (#8100) @evanchaoli :link:

    • Now if opt in EnableCacheStreamedVolumes, worker cache volumes are kept around whilst they are still in use

  • Automatic pipeline archiver archiving child pipelines when parent job has a failed/errored build (#8179) @clarafu :link:

    • Also fixes a bug where the pipeline archiver re-archives pipelines that are already archived

  • Don't run resource type check if within check interval (#8253) @clarafu :link:

    • Fixes a bug where resource types were getting checked every build. Now they will respect the resource checking interval and not run a check if the interval has not elapsed.

  • Fix acrossStep handling for more than 3 vars (#8271) @nexeck :link:

  • Inherite env proxy configure when tls enabled (#8306) @xtremerui :link:

    • Fix a bug that proxy setting through env var got lost when TLS is enabled by --tls-bind-port

  • Introduce OIDC get user info flag (#8353) @danpilch :link:

    • Add CONCOURSE_OIDC_DISABLE_GET_USER_INFO flag. OIDC connector will now fetch additional claims from OpenID UserInfo endpoint. This should fix the problem of configuring Concourse team auth by OIDC user groups due to groups claims missing in some identity providers' auth response.

  • Fix default username prompt for local logins (#8362) @ibokuri :link:

    • Ensure the default username prompt for local logins is properly set.

🤷 Miscellaneous

  • Automatic pipeline archiver accidentally archives child pipelines during a build run (#8137) @clarafu :link:

  • Bump cloudfoundry dependencies in go mod (#8219) @xtremerui :link:

  • UI build page tweaks (#8254) @xtremerui :link:

    • Increase contrast on Build page title when build is aborted. Add more spacing and boarder for history build numbers.

  • Update credhub var lookup logic (#8265) @xtremerui :link:

  • Add Quick Start Documentation for M1 Machines (#8282) @jlamb1 :link:

  • Bump vault api and add srv lookup flag (#8324) @xtremerui :link:

    • Bump Vault API package to latest version and add --disable-srv-lookup flag to Vault configure. If your current Vault URL contains a port number, this change makes no impact. If your Vault URL dose not contain port number, by default SRV lookup is opt-in for backward compatibility. In this case, one can use the flag to disable the feature to avoid unnecessary requests from Vault client.

  • Do not force resource type check on get step (#8363) @xtremerui :link:

📦 Bundled resource types

v7.7.1

2 years ago

🐞 Bug Fixes

  • Fix automatic pipeline archive bug (#8200) @clarafu :link:

    • Removes the line that introduced a bug in 7.7.0 where child pipelines can get archived accidentally.
    • Fixes a bug where pipelines that are already archived get re-archived

🤷 Miscellaneous

  • Backport allow go 1.18 darwin error message in unit tests (#8203) @clarafu :link:

📦 Bundled resource types

v7.7.0

2 years ago

✈️ Features

  • Automatically pause pipelines (#7876) @taylorsilva :link:

    • Adds a new component that will automatically pause pipelines that have not run in more than the configured number of days. The number of days can be configured with CONCOURSE_PAUSE_PIPELINES_AFTER. A value of zero (the default) disables this component. On first run it will retroactively pause pipelines that already fall out of the given day range.
      • Exceptions: Pipelines that have no builds falling within the day range but have been updated in <24hrs will not be automatically paused. This is to cover the case where someone sets a new pipeline and hasn't run any jobs for it yet.

  • Allow task/set_pipeline name to include across step var (#7717) @xtremerui :link:

    • Identifiers for task and set_pipeline steps wrapped by the across step can now have their identifier/step name as a var ((.:some-var)) and won't receive a warning about the name being deprecated

  • Adding missing prometheus emitters for events (#7730) @maxknee :link:

    • Adding GC and missing metrics to prometheus emitter.

  • Adding dry-run mode to fly set-pipeline command (#7734) @Caprowni :link:

    • This adds a dry-run feature to the set-pipeline command within the Fly CLI, the main purpose of this is to allow users to check what would be changed without any interactive-prompt/danger of applying by mistake.

  • Prefer FLY_HOME over HOME (if set) as the directory for storing .flyrc (#7747) @rcw5 :link:

  • Add team to resource commands and archive pipeline command (#7772) @Caprowni :link:

    • Added --team flag to fly command check-resource, you can use it like this fly -t dev check-resource -r some-pipeline/branch:master/myresource --team test
    • Added --team flag to fly command check-resource-type, you can use it like this fly -t dev check-resource-type -r some-pipeline/branch:master/myresource --team test
    • Added --team flag to fly command resources, you can use it like this fly -t dev resources -p some-pipeline --team test
    • Added --team flag to fly command resource-versions, you can use it like this fly -t dev resource-versions -r some-pipeline/branch:master/myresource --team test
    • Added --team flag to fly command archive-pipeline, you can use it like this fly -t dev archive-pipeline --pipeline some-pipeline --team test

  • Batch opentelemetry requests (#7840) @schmurfy :link:

  • AWS SecretsManager can be used from var_sources (#7897) @PG2000 :link:

  • Shared (top level) secrets in AWS secrets manager (#7928) @areller :link:

  • Garbage collect task caches from paused pipelines (#7989) @xtremerui :link:

    • When a pipeline or a job is paused, the task caches that used in the pipeline's job will be garbage collected. This should help free up worker disk space.

  • Add build event for volume streaming (#8031) @andy-paine :link:

    • Build logs will now contain new events when a volume is being streamed to a worker Screenshot 2022-02-02 at 11 39 01

  • Optimize ATC performance by avoid unneccessary go-routines of no-op check notifiers (#8045) @evanchaoli :link:

  • Allow use of fields when using AWS Secrets Manager (#8055) @kurtmc :link:

  • Ignore cached input from volume-locality's consideration (#8061) @evanchaoli :link:

    • When EnableCacheStreamedVolume is enabled and container placement strategy is volume-locality, as get step may not fetch a resource if the resource is found in cache, following step containers may all be placed to the worker where cached resource is found. That worker might be overloaded when there are other workers available. This PR fixes the problem.

  • Enhance volume-locality strategy (#8063) @evanchaoli :link:

    • Optimize database queries for volume-locality container placement strategy

  • Optimize limit active tasks strategy logic (#7874) @xtremerui :link:

    • Optimize limit-active-tasks strategy to reduce DB load and avoid deadlocking when under heavy load.

  • Allow text whitespace to be preserved in step metadata (#8128) @steve-sienk :link:

    • Wrap text of resource metadata on web view.

🐞 Bug Fixes

  • Fix bug of resource type checks (#8048) @evanchaoli :link:

    • Remove lock on checking resource types and prototypes, this will result in a small behaviour change. For example, if you have multiple resources that use the same resource type and those resources run a check for that resource type at the same time, the resource type will be checked multiple times. This was the behaviour before 7.3.0 so we are reverting back to this behaviour.
    • Fixes a bug introduced in 7.6.0 where resource type resource_config_id were never updated.

  • Increase color contrast on build page (#7756) @xtremerui :link:

    • Increase contrast of text on build page after color changes from the previous release that made it harder to read the text

  • Add error message to the create artifact API call (#7899) @Caprowni :link:

  • Porting down migration improvements for build event sequences (#7913) @clarafu :link:

    • If you are on v7.6.0 and want to downgrade, you might end up with some builds that never finish and run into some web log errors like pq: relation "build_event_id_seq_<sequence-id>" does not exist. This is because of a bug in the down migration which is fixed with this PR.
      • If you do run into this problem, you can easily fix it by running create sequence build_event_id_seq_<sequence-id> minvalue 0 start with 0; on your postgres database (You will need to replace with the id that does not exist in the error).

  • Only delete btrfs mounts if *.img exists (#7920) @taylorsilva :link:

    • Concourse worker would fail to start if it's on a btrfs filesystem and tries to use the overlay driver

  • AWS Secrets Manager will treat secret with marked for deletion as deleted (#7933) @areller :link:

  • Handling huge volumes transfer in P2P streaming (#7942) @evanchaoli :link:

    • Fix a bug that P2P streaming would fail if streaming a volume takes longer than 3 minutes. This fix should be applied to both ATCs and workers.

  • Avoid stale prometheus worker metrics when using multiple web nodes (#7965) @databus23 :link:

  • GC builds based on chronological order (#7978) @xtremerui :link:

    • Fix a bug that events of a rerun build be reaped immediately if its prarent build is already reaped. Now candidate builds for GC will be ordered chronologically.

  • Run task caches collector when ATC starts (#7987) @xtremerui :link:

    • Previously when a pipeline is archived, the task caches used in its job will not be garbage collected, which will cause volume leaks in worker disk. Now a component for GC task caches will runs when ATC starts.

  • Bump concourse dex by upstream v2.31.0 (#8050) @xtremerui :link:

    • Fix Bitbucket auth connector failure due to upstream API deprecation.

  • Render nested across steps (#8069) @areller :link:

    • Fix a rendering issue with nested across steps.

  • Fix syntax error in down migration file (#8082) @xtremerui :link:

    • Fix a SQL syntax error that might cause down migration failure.

  • Render build page correctly for legacy aggregate step (#8092) @xtremerui :link:

    • Show legacy builds with aggregate steps. Pipeline configure with aggregate step is still deprecated. This is just fixing the UI rendering error.

  • Fix log line in stream out (#8098) @andy-paine :link:

🤷 Miscellaneous

  • Fix fly hijack due to containerd runc update (#7886) @xtremerui :link:

  • Refactor pipeline lifecycle in atc/db (#7895) @taylorsilva :link:

  • Do not send check build events to syslog drainer (#7922) @xtremerui :link:

    • Since v7.0, resouce and resource type checks are ran as builds. When syslog drainer is enabled, those check build events are also sent to external server, which requires storage space (depends on amount of resources and check interval). Now this type of events will be ignored by syslog drainer.

  • Bump Golang to 1.17 in go.mod and add goproxy in dockerfile (#7926) @evanchaoli :link:

  • Remove unused gc-container-collector-dropped metric (#7932) @databus23 :link:

  • Bump golang crypto lib (#8032) @xtremerui :link:

  • A tiny db sql refactor of changing multiple from to left join (#8046) @evanchaoli :link:

  • Bump worker version to 2.4 (#8081) @xtremerui :link:

📦 Bundled resource types

v7.4.4

2 years ago

🐞 Bug Fixes

  • Fix down migration for build event sequences (#7860) @clarafu :link:
    • This PR fixes a typo in a down migration that affects Concourse version 7.4.1, 7.4.2 and 7.4.3. Therefore, you will be unable to downgrade from any of those listed versions. If you are currently on 7.4.1, 7.4.2 or 7.4.3, you will need to downgrade back to 7.4.0 before you can upgrade to this version of 7.4.4. The instructions on how to downgrade to 7.4.0 is documented in the release notes of the 7.4.x version you are on.

📦 Bundled resource types

v7.4.3

2 years ago

🚨 IMPORTANT!

If you are on 7.4.1 or 7.4.2, you will need to downgrade back to 7.4.0 using backups you have or the script 7.4.x-down-migration.sh included in the assets of this release. Only after downgrading can you safely upgrade directly to 7.4.3 or any later version of Concourse

  • If you are using the 7.4.x-down-migration.sh script to downgrade your Concourse deployment, you do NOT need to run the concourse migrate command because the script essentially is doing that for you. The reason we need this script is because there is a bug in the concourse migrate downgrade scripts. The script requires you to have the psql, access to your postgres database and also the following environment variables that you should already have from starting concourse $CONCOURSE_POSTGRES_PASSWORD, $CONCOURSE_POSTGRES_USER, $CONCOURSE_POSTGRES_HOST, $CONCOURSE_POSTGRES_DATABASE, $CONCOURSE_POSTGRES_PORT.

🐞 Bug Fixes

  • Change migrations timestamp to avoid upgrade issues ( CI: skip-migrations-check ) (#7754) @muntac :link:

    • Change timestamp of backported DB migration to avoid skipping any migrations when upgrading to 7.5 and later versions.
      • If you are on 7.4.1 or 7.4.2 then downgrade back to 7.4.0 using backups you have or the concourse migrate command, using the 7.4.1/7.4.2 version of the binary to downgrade back to 7.4.0. The 7.4.0 database version is 1625844436. Then you can safely upgrade directly to 7.4.3 or any later version of Concourse

  • Fixes for bugs in 7.4.2 (#7758) @clarafu :link:

    • Fix bugs introduced in 7.4.1 related to the errors save image get event: pq: duplicate key value violates unique constraint "pipeline_build_events_x_build_id_event_id" and create resource config: pq: deadlock detected

📦 Bundled resource types

v7.4.2

2 years ago

🚨 IMPORTANT! 🚨

This release contains a bug https://github.com/concourse/concourse/issues/7683. Please avoid upgrading to this version and either upgrade to the next minor (7.6.0) or patch version (7.4.3) If you are on 7.4.1 or 7.4.2, you will need to downgrade back to 7.4.0 using backups you have or the script 7.4.x-down-migration.sh included in the assets of this release. Only after downgrading can you safely upgrade directly to 7.4.3 or any later version of Concourse

  • If you are using the 7.4.x-down-migration.sh script to downgrade your Concourse deployment, you do NOT need to run the concourse migrate command because the script essentially is doing that for you. The reason we need this script is because there is a bug in the concourse migrate downgrade scripts. The script requires you to have the psql, access to your postgres database and also the following environment variables that you should already have from starting concourse $CONCOURSE_POSTGRES_PASSWORD, $CONCOURSE_POSTGRES_USER, $CONCOURSE_POSTGRES_HOST, $CONCOURSE_POSTGRES_DATABASE, $CONCOURSE_POSTGRES_PORT.

🐞 Bug Fixes

  • Fixed a bug where there would be a duplicate key violation for the event ID column in the build_events table (#7736) @clarafu :link:

📦 Bundled resource types

v7.6.0

2 years ago

✈️ Features

  • Make Build page spacing consistent and color theme updated for accessibility (#7497) @ xtremerui :link:
    image

  • Avoid peridoic check build to use db (#7208) @evanchaoli :link:
    Changing Lidar triggered check builds to not use the database, which should mitigate the performance drop introduced by the big refactory of resource checks since 7.0.0.

  • Add default get/put/task timeout (#7426) @evanchaoli :link:

    • Allows Concourse administrator to configure global timeout for get, put and task steps.
    • Fixed a bug where global check timeout didn't work.

  • Indicate if a pipline is archived in pipeline view (#7463) @xtremerui :link:
    When viewing an archived pipeline (or any sub routes of it) in UI, the pipeline name now shows "archived" and the breadcrumbs background will change to grey so one won't confuse.

  • Worker: baggageclaim emits spans (#7487) @taylorsilva :link:

    • Workers now emit traces from the baggageclaim server so one can see volumes being created and streamed as part of a build

🐞 Bug Fixes

  • atc/gc: make build reaper more robust (#7530) @taylorsilva :link:

    • Make build log reaper more robust by not exiting early if it encounters an issue while iterating over pipelines/jobs. Before this change build logs for some pipelines could have accumulated endlessly even with a build retention policy.

  • on_error should not run the hook when err is retriable (#7588) @taylorsilva :link:

    • Fixed a bug when --enable-rerun-when-worker-disappears was enabled and a job/step had an on_error hook. If the step was retried the on_error hook would run when it should not.

  • Revert node selection for rendering boxes (#7616) @taylorsilva :link:

    • Fixed a bug where jobs or resources whose name contained a dot . would not render correctly in the UI

🤷 Miscellaneous

  • Clear resource config scope ID if source field changes (#7494) @taylorsilva :link:

  • Fix small typo on removeAll call comment (#7551) @EstebanFS :link:

  • Bump timeout for volume gc in integration tests (#7591) @clarafu :link:

  • Fix compilation error for bosh topgun test in CI (#7615) @xtremerui :link:

  • goimport all go files (#7661) @taylorsilva :link:

  • Removing un-needed loop (#7696) @maxknee :link:

  • Update test due to in-DB check change (#7707) @xtremerui :link:

  • Change sync.mutex to a pointer (#7719) @taylorsilva :link:

  • Static check resolutions (#7720) @taylorsilva :link:

📦 Bundled resource types

v7.4.1

2 years ago

🚨 IMPORTANT! 🚨

This release contains a bug https://github.com/concourse/concourse/issues/7683. Please avoid upgrading to this version and either upgrade to the next minor (7.6.0) or patch version (7.4.3) If you are on 7.4.1 or 7.4.2, you will need to downgrade back to 7.4.0 using backups you have or the script 7.4.x-down-migration.sh included in the assets of this release. Only after downgrading can you safely upgrade directly to 7.4.3 or any later version of Concourse

  • If you are using the 7.4.x-down-migration.sh script to downgrade your Concourse deployment, you do NOT need to run the concourse migrate command because the script essentially is doing that for you. The reason we need this script is because there is a bug in the concourse migrate downgrade scripts. The script requires you to have the psql, access to your postgres database and also the following environment variables that you should already have from starting concourse $CONCOURSE_POSTGRES_PASSWORD, $CONCOURSE_POSTGRES_USER, $CONCOURSE_POSTGRES_HOST, $CONCOURSE_POSTGRES_DATABASE, $CONCOURSE_POSTGRES_PORT.

✈️ Features

  • 7.4.x: add some DB optimizations (#7641) @taylorsilva :link:
    • DB optimizations
      • Increment an event ID in-memory instead of using a Postgres Sequence
      • Only update resource config if it hasn't been updated in a minute
      • Close zstd reader in load var step

🤷 Miscellaneous

  • Backport baggageclaim to release/7.4.x (#7432) @taylorsilva :link:

📦 Bundled resource types

v7.5.0

2 years ago

IMPORTANT: This release includes a few large refactors, therefore we recommend anyone using Concourse for mission-critical workflows to wait for the next few releases just in case any edge cases are found.

This release contains the version of github-release-resource that fixes the illegal base64 data at input errors from GitHub API's recent breaking change https://github.com/concourse/github-release-resource/issues/108.

🚨 Breaking

  • Prefer overlay over btrfs in baggageclaim when using driver: detect (#7427) @aoldershaw :link:

    • Previously, when the baggageclaim driver was not specified, Concourse attempts to detect the supported drivers
    • The prior driver precedence is: btrfs -> overlay -> naive
    • The new driver precedence is: overlay -> btrfs -> naive

  • Allow team members to archive pipelines (#7449) @wanderanimrod :link:

    • Users with the member role on a team can now archive pipelines by default. The "archive pipeline" action was previously assigned to the owner role. If you've configured your own RBAC this change will not effect you.

✈️ Features

  • Removing VersionedResourceTypes from get, check and put plans (#7176) @clarafu :link:

    • This PR is mainly a refactor but there is a behavioural change that comes along with it. If a resource uses a custom resource type, its Put, Check, Get and Task steps will now always create a check for its parent custom resource type. This check will still respect the resource checking interval and the check_every of the resource type. Because of this new feature, we no longer need to explicitly check the custom resource types in lidar.

  • Add audit information for job & pipeline pauses (#7273) @rjinskidepop :link:

    • Add pipeline and job pause meta information - who and when.

  • Propagate groups between subpages of a pipeline (#7307) @clarafu :link:

    • If a user was initially viewing a group in the pipeline page, this will be persisted in the pipeline breadcrumb when navigating between pipeline subpages.

  • Optimize pipeline svg rendering (#7438) @aoldershaw :link:

    • The initial render of the pipeline page should be much faster, particularly on Chrome 92+

  • Optimize build log collection (#7327) @evanchaoli :link:

    • Optimized a SQL statement used to remove build logs. This optimization will specially benefit large deployments that have a lot of pipelines.

  • Don't query for the entire resource in the check delegate (#7474) @aoldershaw :link:

  • Simplify atc/worker package and extract runtime abstractions (#6597) @aoldershaw :link:

    • We will now error when a suitable worker does not exist rather than waiting forever.

🐞 Bug Fixes

  • Fix opening a link to a resource causality page (#7369) @aoldershaw :link:

  • Don't allow empty identifiers when renaming pipelines/teams (#7370) @taylorsilva :link:

    • Fixed a bug where a pipeline or team could be renamed to an empty string. The team/pipeline could not be deleted through fly. An error is now returned by the API if the identifier is blank

  • Sanitize prometheus metric labels (#7423) @lrstanley :link:

    • Ensure Prometheus metric labels are valid. This resolves an issue with our bosh release, where web nodes would fail to start, due to a metric label that wasn't valid according to Prometheus.

  • Fix overlapping between inputs and jobs in UI (#7454) @xtremerui :link:

    • Fixes an edge case that might overlap an input and job node in the pipeline view.

  • Validate if a Pipeline contains a cycle (#7455) @EstebanFS :link:

    • The API will reject any pipelines that contains a cycle

  • Prevent open redirect to other hosts (#7459) @taylorsilva :link:

    • Prevent an open redirect vulnerability on the /sky/login path

  • Delete btrfs volume if it exists when using the overlay driver (#7461) @taylorsilva :link:

    • Made worker initialization more stable if you're switching from btrfs to overlay. The worker will remove the btrfs mount if it exists before creating overlay mounts

  • Fix missing label in metric concourse_steps_waiting (#7479) @Esysc :link:

    • Fix missing label in metric concourse_steps_waiting

  • Close zstd reader in load var step (#7548) @clarafu :link:

🤷 Miscellaneous

  • Speed up TSA tests (#7336) @aoldershaw :link:

  • Don't mount Concourse source directory in integration tests (#7347) @aoldershaw :link:

  • Add baggageclaim to concourse/concourse (#7351) @taylorsilva :link:

  • Bump opentelemetry package to 1.0.0-RC2 (#7360) @taylorsilva :link:

  • Remove experimental warning for set_pipeline and load_var steps (#7396) @aoldershaw :link:

  • Skip integration downgrade test (#7398) @aoldershaw :link:

  • Skip integration upgrade test (#7401) @aoldershaw :link:

  • namespace prometheus test to avoid conflict (#7436) @xtremerui :link:

  • Fix integration upgrade/downgrade tests (#7450) @aoldershaw :link:

  • Fix DB performance regression with updating resource cache metadata (#7472) @aoldershaw :link:

  • Don't create workdir volume for check steps (#7473) @aoldershaw :link:

📦 Bundled resource types