Google Caliban Versions Save

Incremental release for JOSS, plus:

• Move cloud_sql_proxy installation before code copy (https://github.com/google/caliban/pull/87).

The biggest feature in this new release is native support for logging to an MLFlow tracking server using the UV Metrics project, via https://github.com/google/caliban/pull/35) This feature is in alpha, and baking internally; expect documentation soon!

If you want to try this out, check out the tutorial in tutorials/uv-metrics: https://github.com/google/caliban/blob/master/tutorials/uv-metrics/README.md

More features

• minor bugfixes for GKE (https://github.com/google/caliban/pull/85)
• additional tests for gke.{types, util} (https://github.com/google/caliban/pull/84)
• re-order custom apt packages before pip requirements (https://github.com/google/caliban/pull/82)
• modify base image to our more general cloudbuild naming scheme (https://github.com/google/caliban/pull/80)
• updated google-auth dependency version to 1.19.0 (https://github.com/google/caliban/pull/79)
• add clearer contribution info (https://github.com/google/caliban/pull/76)
• Update uv-metrics tutorial (https://github.com/google/caliban/pull/74, https://github.com/google/caliban/pull/72)
• add support for running an embedded cloudsql_proxy (https://github.com/google/caliban/pull/60)
• bugfix for #65: do not add resource maxima when quota is < 1 (#67)
• Updated accelerator regions (and globally availabe AI Platform regions to match the current state here): https://cloud.google.com/ai-platform/training/docs/regions

This release was focused on making it easier for others to contribute. The highlights are:

• we now support custom base images
• you no longer need a service account key to submit jobs to AI Platform
• we won't push anymore if a docker image already exists in Cloud, saving you some time and stdout spray
• custom base images via .calibanconfig.json, AND support, with special base image names, for all of Google's "Deep Learning VMs" instead of Caliban's default base images.

Thanks to @ramasesh , @eschnett, @ajslone and @sagravat for their contributions on this release!

For more detail, here's the CHANGELOG:

• @ramasesh Added a fix that prevented pip git dependencies from working in caliban shell mode (https://github.com/google/caliban/pull/55) This adds a small update to the base image, so be sure to run

docker pull gcr.io/blueshift-playground/blueshift:cpu
docker pull gcr.io/blueshift-playground/blueshift:gpu

to get access to this fix.

• Thanks to @eschnett, --docker_run-args can now deal with arbitrary whitespace in the list of arguments, instead of single spaces only. (https://github.com/google/caliban/pull/46)

• Caliban now authenticates AI Platform job submissions using the authentication provided by gcloud auth login, rather than requiring a service account key. This significantly simplifies the setup required for a first time user.

• caliban cloud now checks if the image exists remotely before issuing a docker push command on the newly built image (https://github.com/google/caliban/pull/36)

• Big internal refactor to make it easier to work on code, increase test coverage, add new backends (https://github.com/google/caliban/pull/32)

• add schema validation for .calibanconfig.json. This makes it much easier to add configuration knobs: https://github.com/google/caliban/pull/37

• Custom base image support (https://github.com/google/caliban/pull/39), thanks to https://github.com/google/caliban/pull/20 from @sagravat. .calibanconfig.json now supports a "base_image" key. For the value, can supply:

  • a Docker base image of your own
  • a dict of the form {"cpu": "base_image", "gpu": "base_image"} with both entries optional, of course.

  Two more cool features.

  First, if you use a format string, like "my_image-{}:latest", the format block {} will be filled in with either cpu or gpu, depending on the mode Caliban is using.

  Second, we now have native support for Google's Deep Learning VMs as base images. The actual VM containers live here. If you provide any of the following strings, Caliban will expand them out to the actual base image location:

dlvm:pytorch-cpu
dlvm:pytorch-cpu-1.0
dlvm:pytorch-cpu-1.1
dlvm:pytorch-cpu-1.2
dlvm:pytorch-cpu-1.3
dlvm:pytorch-cpu-1.4
dlvm:pytorch-gpu
dlvm:pytorch-gpu-1.0
dlvm:pytorch-gpu-1.1
dlvm:pytorch-gpu-1.2
dlvm:pytorch-gpu-1.3
dlvm:pytorch-gpu-1.4
dlvm:tf-cpu
dlvm:tf-cpu-1.0
dlvm:tf-cpu-1.13
dlvm:tf-cpu-1.14
dlvm:tf-cpu-1.15
dlvm:tf-gpu
dlvm:tf-gpu-1.0
dlvm:tf-gpu-1.13
dlvm:tf-gpu-1.14
dlvm:tf-gpu-1.15
dlvm:tf2-cpu
dlvm:tf2-cpu-2.0
dlvm:tf2-cpu-2.1
dlvm:tf2-cpu-2.2
dlvm:tf2-gpu
dlvm:tf2-gpu-2.0
dlvm:tf2-gpu-2.1
dlvm:tf2-gpu-2.2

Format strings work here as well! So, "dlvm:pytorch-{}-1.4" is a totally valid base image.

This release:

• Prepared for a variety of base images by setting up a cloud build matrix: https://github.com/google/caliban/pull/25
• Added better documentation for gcloud auth configure-docker https://github.com/google/caliban/pull/26
• Added close() to TqdmFile, preventing an error when piping stdout: https://github.com/google/caliban/pull/30
• tqdm progress bars and other interactive outputs now display correctly in caliban run outputs. stdout flushes properly! Before these changes, stderr would appear before any stdout, making it difficult to store the logs in a text file. Now, by default, python processes launched by caliban run won't buffer. https://github.com/google/caliban/pull/31

Thanks to @dthiagarajan for reporting an issue that led to this work.

This release:

• fixes the python binary that caliban notebook points to (now that we use conda)
• adds DEBIAN_FRONTEND=noninteractive to the apt-get command, so that packages like texlive won't freeze and wait for you to specify a timezone.

This makes it easy to add, for example, npm and latex support to your caliban notebook invocations.

Enjoy!

google-auth pushed an upgrade that broke an internal method we were using. This release pins the version and fixes the error.