Bazel Remote Cache + Remote Execution in Kubernetes
Bazel | REAPI | Kubernetes |
buildkube uses rules_docker and rules_k8s to build and deploy bazel-buildfarm (java), bazel-buildbarn (golang) and/or buildgrid (python) into an existing kubernetes cluster. These are the 3 known open-source server-side implementations of the remote-execution-api (REAPI), plus the closed source google Remote Build Execution (RBE) service (alpha).
Known clients of the REAPI include bazel itself, recc, and possibly pants.
WORKSPACE
file k8s_defaults
rule to point to your kubernetes
cluster (should match $ kubectl config current-context
)$ (cd farm/ && make install)
$ (cd farm/ && make port-forward)
$ make abseil_clone
$ make abseil
$ bazel test //...
.$ kubectl get pods --all-namespaces
to see all.replicas
in the deploy.yaml
files and/or bazelrc
file.BuildFarm worker does not detect if server goes down. Must manually kubectl delete pod --selector=k8s-app=worker
when re-installing or updating server
deployment.
When a worker registers itself with the server (operation-queue), it provides
a dict of key:value pairs that must match the action execution requirements.
In particular, the worker.config
container-image
key MUST be exactly
matching the rbe_ubuntu image tag.
/tmp/abseil-cpp/absl/utility/BUILD.bazel:22:1: C++ compilation of rule '//absl/utility:utility_test' failed (Exit 34). Note: Remote connection/protocol failed with: execution failed catastrophically
.NOTE(@EdShoueten): There are three ways that can be used to alleviate this issue:
- Spawn more workers on your cluster.
- Pass in an explicit --jobs= to the build that is the same order of magnitude as the number of workers.
- Tune this flag on the scheduler process: https://github.com/EdSchouten/bazel-buildbarn/blob/master/cmd/bbb_scheduler/main.go#L22
main
) must match across the bazelrc
--instance_name=main
,
server args -scheduler main|ubuntu-scheduler:8981
, and worker args bot --remote=http://server:8980 --parent=main host-tools