Showcase of how to manage building projects inside monorepo with Gradle as build tool and CircleCI, Bitbucket Pipelines, Travis CI or GitHub Actions as CI tool.
This is an example of how to manage building projects inside monorepo with Gradle as build tool and one of the following services as CI tool:
Service | Status |
---|---|
CircleCI | |
Bitbucket Pipelines | |
Travis CI | |
GitHub Actions |
When I push some changes to monorepository I want to
There is only one main job called build started automatically on every push. This job is responsible for triggering another jobs for each affected project in order with respecting project dependencies.
Build job is running until all triggered jobs are finished.
Build job is successful only when there were no failed jobs (even when there were no jobs).
There is file tools/ci/projects.txt
which contains lines with glob patterns pointing to root directories of all supported projects.
Jobs are defined in default location depending on which CI tool you are using.
.circleci/config.yml
bitbucket-pipelines.yml
.travis.yml
.github/workflows/
Currently there is a convention used for mapping project to CI job. Job name is resolved from project's directory path as last path component.
e.g. project under directory
apps/server
is built by jobserver
.
Dependencies are based on Gradle's composite build feature. To define dependency between projects use includeBuild
function in project build script (usually in settings.gradle
).
If you are not using Gradle or you want to add some additional dependency you can specify it in dependency file. Default location is in each project directory on path .ci/dependencies.txt
. Location can be changed by setting environment variable CI_DEPENDENCIES_FILE
. File should contain lines with paths to other projects (relative paths to monorepo root, e.g. libs/common).
To respect dependencies between projects jobs are triggered in multiple rounds. For each round one or more jobs are triggered and only when all jobs are successfully finished next round is processed. Even if there is only one failed job all next rounds are skipped and whole build is failed.
Commit message can contain some special words which if found can modify default building behavior.
Supported commands:
Whole logic is implemented as bunch of Bash scripts under directory tools/ci
. Every script can be run directly and it should output some documentation when it requires some parameters.
Implementation is split to core logic and plugins for different CI tools. Core logic can be found under tools/ci/core
and plugins are under tools/ci/plugins
.
Main script is tools/ci/core/build.sh
and it is only thing started from build job.
There is tool called jq used for JSON parsing.
You need to care only when you want run it locally because jq is usually part of default images used in CI tools.
It is possible to run tools/ci/core/build.sh
locally but there is need to provide few environment variables depending on CI tool used.
CI_TOOL=circleci \
CIRCLE_API_USER_TOKEN=XXX \
CIRCLE_PROJECT_USERNAME=zladovan \
CIRCLE_PROJECT_REPONAME=monorepo \
CIRCLE_BRANCH=master \
CIRCLE_SHA1=$(git rev-parse HEAD) \
tools/ci/core/build.sh
Where:
Note that this command could trigger some jobs in circleci
CI_TOOL=bitbucket \
BITBUCKET_USER=zladovan \
BITBUCKET_PASSWORD=xxx \
BITBUCKET_REPO_FULL_NAME=zladovan/monorepo \
BITBUCKET_BRANCH=master \
BITBUCKET_COMMIT=$(git rev-parse HEAD) \
tools/ci/core/build.sh
Where:
Note that this command could trigger some jobs in bitbucket
CI_TOOL=travis \
TRAVIS_TOKEN=xxx \
TRAVIS_REPO_SLUG=zladovan/monorepo \
TRAVIS_BRANCH=master \
TRAVIS_COMMIT=$(git rev-parse HEAD) \
tools/ci/core/build.sh
Where:
Note that this command could trigger some jobs in travis
CI_TOOL=github \
GITHUB_REPOSITORY=zladovan/monorepo \
GITHUB_TOKEN=xxx \
GITHUB_REF=refs/head/master \
GITHUB_SHA=$(git rev-parse HEAD) \
tools/ci/core/build.sh
Where:
Note that this command could trigger some jobs in github
Folder structure used in this repository is only an example of how it can look like. It is possible to use any structure, there is only need to use different patterns in tools/ci/projects.txt
apps/
└── stand-alone runnable and deployable applications
libs/
└── reusable libraries (used in apps dependencies)
tools/gradle-plugins/
└── reusable gradle logic (used in apps and libs builds)
tools/ci
└── ci scripts
realpath
usedThanks to Tufin for inspiration in Tufin/circleci-monorepo.