Spring Cloud Pipelines Versions Save

[DEPRECATED] Codebase containing Concourse and Jenkins opinionated pipelines. Moved to https://github.com/CloudPipelines/

v1.0.0.M9

5 years ago

Spring Cloud Pipelines

Spring Cloud Pipelines is a GitHub project that tries to solve the following problems:

  • Creation of a common deployment pipeline
  • Propagation of good testing and deployment practices
  • Reducing the time required to deploy a feature to production.

The first commit took place on 31-08-2016. Since then, we have gotten a lot of feedback from the community related to the suggested deployment pipeline and its concrete visualisations. Over those two years, the most important features we have managed to build are:

  • Opinionated deployment pipeline setup
  • Scripts for the pipeline, to verify backward compatibility of your project and allow zero-downtime deployment
  • Support for PHP, .NET, NodeJS, and JVM (Maven & Gradle) projects
  • Deployment option for Cloud Foundry
  • Deployment option for Kubernetes
  • Pipeline visualisation in Jenkins by using Jenkins Job DSL
  • Pipeline visualisation in Jenkins by using Jenkinsfile
  • Pipeline visualisation in Concourse

We would like to announce the next 1.0.0.M9 release of Spring Cloud Pipelines, which will also be its last in the current form.

(Spring) Cloud Pipelines

Since I'm a member of the Spring Cloud team, we have decided to put it under the Spring Cloud umbrella. Other than that, the project has nothing to do with Spring Cloud, not to mention Spring in general.

That is why we have decided to split the project into pieces, rename it, and put it into a separate GitHub organization.

Welcome, Cloud Pipelines!

Spring Cloud Pipelines has a new home and a new name. The projects now lays under the https://github.com/CloudPipelines/[Cloud Pipelines] organisation at GitHub. The project, as a result, was split into the following pieces:

  • Scripts: Contains core logic of the deployment pipeline. You can call this repository recipes. Each recipe contains a step of the deployment pipeline. The documentation is here.
  • Concourse: Pipeline visualisation done with Concourse. The documentation is here.
  • Jenkins: Pipeline visualisation done with Jenkins. The documentation is here.
  • Project Crawler: Abstraction over fetching data from repository management services.
  • Pipeline Base: Docker base image used by CI servers.

After migrating the projects to the new organisation, we have decided to mark them as feature complete. Of course, we will facilitate any discussions around the project and chores related to issues, pull requests and releases, but we can safely state that the project will be fully driven and maintained by the community.

If you are interested in migrating your project from Spring Cloud Pipelines to Cloud Pipelines, you should check out the migration guides:

Why Spinnaker?

Spinnaker is an open source multi-cloud continuous delivery platform which started at Netflix but now involves a wider community of contributors from Google, Amazon, Pivotal, and many others. A collaborative effort by the Spring team and Pivotal Cloud R&D has led to Cloud Foundry support in Spinnaker.

Spinnaker gives us access to a wider array of supported cloud providers without having to reinvent these interactions.

The work to-date on zero-downtime deployments and rollback testing in Cloud Pipelines reveals a key value of a continuous delivery platform like Spinnaker that separately maintains an inventory of the assets in your deployed environments.

  • In Cloud Pipelines, the current production system state was not being stored. To mitigate this issue we used the Git tags to store state. That, however, can be easily broken when manual deployments or rollbacks take place. Also, the production system state may span multiple versions (across cloud providers or regions) and is in general irreducible to a single value that can be stored in a tag. A Spinnaker pipeline performing, for example, a rollback test is free to use Spinnaker’s knowledge of the state of the system to contextualize the rollback test to the production version in the target regions independently of one another.

  • The deployment options provided in Cloud Pipelines are limited. Spinnaker generalizes blue/green, rolling push, zero-downtime and automated canary analysis deployment features across a range of supported cloud providers.

To be more concrete, consider blue/green deployment. Spinnaker supports blue/green while maintaining an arbitrary number of prior version server groups. From the current version of the app in the below example, you are one action away from rolling back to a version of the app that is several versions from the most recent. At any time, we could choose to destroy V022, effectively making it unavailable to roll back to. Such a thing could happen out-of-band from a deployment activity (such as to free capacity), making stateless systems less able to deal with change.

Get Involved!

We welcome your input, in a variety of forms. If you have questions about Spinnaker, ask on Stack Overflow and use the #spinnaker tag. In case of Cloud Pipelines, ask questions on GitHub. If you want to work on the code, we love pull requests. If you spot a problem, please submit an issue on Github.

v1.0.0.M8

5 years ago

This release adds a lot of new features and quality and testing improvements. As for main features, we've added support for a project with multiple modules, and for mono-repos (projects with standalone application sources in it). As for quality and testing, we're approaching 200 Bash tests (we've written missing tests for the Concourse pipeline). We've started publishing the project's sources as a downloadable artifact. That way instead of cloning the repo or downloading the ~25mb archive, you can fetch around 150kb of sources. The docs for 1.0.0.M8 are available here - http://cloud.spring.io/spring-cloud-static/spring-cloud-pipelines/1.0.0.M8/

We're attaching the distributions to this release in tar.gz and zip formats, with sources only.

Below you can find more detailed release notes:

Features

  • Latest prod tag could change while pipeline is running [#125]
  • Don't log any URLs cause it can contain the password [#121]
  • Parametrize the manifest.yml filename [#117]
  • Change the CF way of doing prod deployment [#111]
  • For CF, PAAS_TEST_SPACE should be PAAS_TEST_SPACE_PREFIX [#126]
  • Allow multiple ports for stub runner on CF [#123]
  • Allow passing of credentials to upload to repo with binaries [#116]
  • Allow to reuse existing services for test environment [#114]
  • Utilize the keyval resource in the concourse pipeline [#120]
  • Get last prod-tagged release for back-compatibility and rollback jobs [#124]
  • Add switch to delete services in CF [#112]
  • Separate repo with binaries URL for upload and download [#115]
  • Enabled tag deletion in prod-rollback job [#128]
  • Added parametrization of sc-pipelines.yml file in pipelines [#127]
  • Bumped Gradle to 4.8 [#154]
  • Migrate to use new bats repo [#155]
  • [Jenkins] Download the pipeline definition instead of cloning it from Git [#146]
  • For a release produce an artifact with sources only to make it weigh less [#142]
  • Custom Git commands in commons scripts don't use SSH key [#145]

Breaking changes

  • App manifest is ignored [#129]
    • main source of truth for CF deployment is the manifest
  • Refactor the prod deployment function names [#113]
    • deployGreen or switchBackToBlue functions got removed
  • Add support for multi module/project built applications [#109]
    • the git tag is changed to contain the project name in it. E.g changed from dev/1.0.0.M1 to dev/projectName/1.0.0.M1

Bug fixes

  • Rollback smoke test run the latest tests and not the old ones [#118]
  • The gitSsh credentials in Jenkins are not created [#137]
  • Unit tests for project customization fail when run on Windows OS [#134]
  • Jenkins pipelines generated by sample seed pipeline don't refer Git SSH credentials [#138]
  • Scripts tests fail when specifying PIPELINE_DESCRIPTOR [#142]

Documentation fixes

  • Update the docs with step by step to convert your project [#30]
  • Fixed paragraph about Kubernetess wrongly mentioned Cloud Foundry [#152]

Thanks

Super special thanks go to the one and only @ciberkleid for all her hard work on the project (look at that!!! - https://github.com/spring-cloud/spring-cloud-pipelines/pull/130/) and to @oiavorskyi (https://github.com/spring-cloud/spring-cloud-pipelines/pull/148) Kudos Cora and Oleg!

Special thanks go to @regevbr and everybody involved in the project!

v1.0.0.M7

6 years ago

This release adds a lot of new features and quality and testing improvements. As for main features, we've added Kubernetes support for Jenkins and Concourse. As for quality and testing, we've written over 150 Bash tests, added Shellcheck, EditorConfig and written more Jenkins tests. We've also unified the project's documentation that now can be available for each milestone under Spring Cloud Static gh-pages. The docs for 1.0.0.M7 are available here - http://cloud.spring.io/spring-cloud-static/spring-cloud-pipelines/1.0.0.M7/

Below you can find more detailed release notes:

Features

  • Added Kubernetes support for Jenkins [#86][#63][#87]
  • Add support for SSH based Git access from JenkinsFile Groovy script [#88]
  • Project Gradle setup contains all the required Jenkins dependencies, also for init.groovy [#72]
  • Added EditorConfig [#78]
  • Added Shellcheck [#79][#81][#82][#89]
  • Caching Concourse Maven / Gradle folders [#77][#68]
  • Added Bats tests [#84]
  • Added an option to pass script customizations [#83]
  • Script helpers are more generic [#90][#93]
  • Maven credentials are injectable [#96]
  • Customization of CF, K8S binaries [#102]
  • Add support for specifying application Git branch [#101]
  • Add support for SSH based Git access from JenkinsFile Groovy script [#98]
  • Added project customizer [#55]

Breaking changes

  • Change the pipeline.yml to sc-pipelines.yml [#76]

Bug fixes

  • TOOLS_BRANCH should be used by the seed job [#73]
  • Upgraded Artifactory image [#69]

Refactoring

  • Updated test samples [#75][#74]
  • Updated Concourse structure [#91][#100]
  • Changed the rollback step name [#97]
  • Creation of separate Jenkins seeds [#99]

Documentation fixes

  • Upgraded the way docs are built [#85]

Thanks

Special thanks go to @wybczu, @malston , @bijukunjummen, @anwarchk and everybody involved in the project!

v1.0.0.M6

6 years ago

Features:

  • Added pipeline.yml where you define necessary infrastructure applications to be deployed for tests [#71][#66]
  • Make the stub runner work with embedded stubs enhancement [#35]
  • Added option to customize buildpack [#57]

Bug fixes:

  • Add script for removing prod tags from sample repos enhancement [#64] by @wybczu
  • Fix remove-prod-tags.sh script enhancement [#65] by @wybczu
  • Wrong mysql name is picked for stage bug [#62]
  • API compatibility check fails if no tests are found bug [#59]
  • Other bug fixes [#58][#59]

Refactoring:

  • Initial work by @nebhale on refactoring the Concourse pipeline [#70]
  • Removed any notion of Cloud Foundry and demo applications from root scripts

Documentation fixes:

  • Explain in the docs what the demo apps look like [#67]

v1.0.0.M5

6 years ago

With 1.0.0.M5 we've added a new step in the Build stage called API Compatibility check. Now, your pipeline can break when, as a HTTP / messaging producer, you're doing a backward incompatible change. We're checking what were the latest contracts you've defined for your most recent production deployment and from those contracts we're generating tests to your current application.

Also @ryanjbaxter has added support for GPG keys.

v1.0.0.M4

7 years ago

in this release we've added:

v1.0.0.M3

7 years ago

in this release we've added:

  • blue green production deployment steps
  • a bunch of documentation fixes
  • a couple of issues were fixed related to working with Cloud Foundry

1.0.0.M1

7 years ago

1.0.0.M2

7 years ago

now we also support Gradle projects out of the box; also minor fixes were applied

  • updated docs
  • fixed missing env passing to stub runner containing repo with jars
  • updated scripts used to remove routes