Kubernetes Rust Kubelet
Krustlet v1.0.0 is the first alpha release of 1.0! This marks the the start of API and feature stability to Krustlet. For more information on our roadmap to 1.0, please read our recently released blog post. These release notes contain all breaking changes and new features from 0.7 and will be released with the final 1.0 release as well (for those who go straight from 0.7 to the final 1.0).
As we move towards 1.0, backwards compatibility and API stability will now be respected. However, during the alpha-beta-RC period, there may be times where we will need to break the API. To be clear, here are the guarantees for each type of release:
krator has now been moved to its own repo so it can continue to evolve as its own project! This also means that the kubelet crate now consumes krator directly from crates.iooci-distribution now supports DockerHub authenticationkubelet crate is now instrumented with the tracing crateProvider implementations can now implement an optional shutdown method for graceful shutdown behaviorkrustlet-wasi binary
tracing crate, the log output is significantly different than 0.7.0Rust API
Ref type has been replaced with the new VolumeRef enum. Each variant wraps the logic of each supported volume type while still allowing access to the underlying type. Any VolumeRef can be mounted or unmounted. This should be a cleaner experience for anyone leveraging this part of the APIVolumeType has been removed from the API as part of the VolumeRef workProvider::ProviderState associated type now has 2 additional bounds required:Before:
type ProviderState: 'static + Send + Sync;
After:
type ProviderState: 'static + Send + Sync + PluginSupport + DevicePluginSupport;
These new traits are used to require implementation of plugin registries and device plugin support. A minimal implementation (that does nothing) is below:
struct ProviderState;
impl PluginSupport for ProviderState {}
impl DevicePluginSupport for ProviderState {}
Please note that these traits are also now required for implementors of GenericProvider.
After many discussions, we have decided that CNI and 100% native Kubernetes network support will not be part of the 1.0 release. Networking implementations vary wildly between different Krustlet providers and it would be difficult and unwise to try and define a common abstraction at this time. For example, wasmCloud connects using a system called a “lattice” which can consist of an arbitrary graph of connected hosts. Whereas the CRI provider relies on the CNI spec implementations built in to many Kubernetes distros. Not only are we at a point where we need to understand how more complex networking should work, but the same discussions involve things like sidecars, service meshes, and so on. As a result, we decided to let you handle your own networking here, which you can do, while the community understands what's the best path forward with these more complex interactions.
Our hope is that once we have several different networking examples from various providers, it will be easier to add in a common abstraction at this time. Waiting on that information would only introduce additional delay for releasing Krustlet as an otherwise stable and production-ready (though still bleeding edge) project.
Our next anticipated version is 1.0.0-alpha.2 after we finish the remaining tasks in the 1.0 milestone
We want to express a huge thanks to all of those in the community who have helped the project evolve to this point. We appreciate your efforts in making this project a success.
If we missed you, let us know!
Download Krustlet 1.0.0-alpha.1:
Check out our installation docs for information on how to install Krustlet.
Krustlet v0.7.0 is mainly an SDK focused release. The biggest under the hood change was the update of our underlying task runtime (Tokio) to its first stable release. There are also a few more small SDK feature adds. We have also switched out the logging implementation so that it can eventually be used to pass tracing data to another service. Lastly, this release marks the official move of the wasmCloud Provider to its own repo! For more details on what isn't implemented yet, see the Known Issues section.
Because this is pre-release software, there are no backwards compatibility guarantees for the Rust API or functionality. However, we will do our best to document any breaking changes in future releases.
Please note that this is NOT production-ready software, but it is in a usable state. The WASI standard and wasmtime are still under heavy development, and because of this there are key features (like networking) that are missing; these will appear in the future. However, there is networking support available in the wasmCloud provider.
tracing crate. We will be instrumenting more of our code over the next releaseStatusBuilder
oci-distribution, kubelet, and krator) are now using the Tokio 1.X runtime. This means it will be a breaking change for SDK consumers that are using tokio 0.2 or 0.3 in their codeOur next anticipated version is 1.0.0-alpha.1 (although we will cut a 0.7.1 if necessary). As we move to 1.0 our focus will be on fixing all outstanding bugs and polishing up the CSI support. We also are hoping to streamline the installation and deployment process as much as possible.
After many discussions, we have also decided that CNI and 100% native Kubernetes network support will not be part of the 1.0 release. Networking implementations vary wildly between different Krustlet providers and it would be difficult and unwise to try and define a common abstraction at this time. For example, wasmCloud connects using a system called a "lattice" which can consist of an arbitrary graph of connected hosts. Whereas the CRI provider relies on the CNI spec implementations built in to many Kubernetes distros. However, what we will have is basic host port (and probably node port) support for each of the Wasm based providers. Any provider will be in charge of creating their own networking implementation tailored to the needs of their provider. Our hope is that once we have several different networking examples from various providers, it will be easier to add in a common abstraction at this time. Waiting on that information would only introduce additional delay for releasing Krustlet as an otherwise stable and production-ready (though still bleeding edge) project.
We want to express a huge thanks to all of those in the community who contributed to this release. There have been quite a few bug fix contributions from the community that have been very helpful. We appreciate your efforts in making this project a success.
Download Krustlet 0.7.0:
Check out our installation docs for information on how to install Krustlet.
Krustlet v0.6.0 has several major new features, particularly around alpha level support for Container Storage Interface volumes. There were also some new SDK features (and a new crate!) that necessitated a few more breaking API changes. These changes are explained below. For more details on what isn't implemented yet, see the Known Issues section.
Because this is pre-release software, there are no backwards compatibility guarantees for the Rust API or functionality. However, we will do our best to document any breaking changes in future releases.
Please note that this is NOT production-ready software, but it is in a usable state. The WASI standard and wasmtime are still under heavy development, and because of this there are key features (like networking) that are missing; these will appear in the future. However, there is networking support available in wasCC.
krator! This crate allows for generic reuse of the state machine logic to write any type of Kubernetes controller. Please read the introduction blog post for more informationrun_to_completion method provided by the Kubelet crate, and are spawned from within the Pod state machines of the providers. This is not required to implement a Provider, but we found that it significantly simplified our implementation.We have implemented many common Pod states such that you can borrow one or more of these state handlers from the Kubelet crate rather than rewriting this boiler plate. These states currently include Registered, ImagePull, ImagePullBackoff, VolumeMount, CrashLoopBackoff, and Error.
If you would like to make use of these states, you must implement a few new traits:
GenericProviderState for your shared state.GenericPodState for your object state.GenericProvider for your Provider type.Please refer to either wascc-provider or wasi-provider for examples of how to implement these traits.
We had to make several small changes to the Provider trait to allow for generic state support.
The first is the addition of a new required method and associated type that captures the state of the provider itself (e.g. the container store and handles to running pods) that is shared across all pods:
type ProviderState: 'static + Send + Sync;
fn provider_state(&self) -> crate::state::SharedState<Self::ProviderState>;
SharedState<_> is simply a type alias for Arc<tokio::sync::RwLock<_>>, so you can refer to the Tokio documentation to understand its API.
The second is a change to the associated type for the PodState. This must now be something that implements the ObjectState type from the new krator crate:
type PodState: ObjectState<
Manifest = Pod,
Status = PodStatus,
SharedState = Self::ProviderState,
>;
The last change is the addition of a new method that supports the Kubelet plugin registry (used for registering CSI plugins). This is an optional feature that has a default implementation returning None. If you want to opt in to CSI volumes, you can provide your own implementation of this function
fn plugin_registry(&self) -> Option<Arc<PluginRegistry>>;
All providers will need to implement these new fields and methods (with the exception of plugin_registry) upon upgrading to Krustlet 0.6. You can see an example of how these are implemented in the wasi-provider
The prelude in kubelet::state has been moved to two separate preludes, kubelet::pod::state and kubelet::container::state, which export the same state machine API types but different status types, etc.
If you are not using the prelude, please be aware that a number of types were moved from kubelet::state to krator::state.
The State trait has changed in a few ways:
In the next method, a new argument has been introduced for accessing shared state: shared: SharedState<ProviderState>.
The next method now takes Manifest<Pod> instead of Pod. This wrapper allows access to a reflection of the Pod rather than a potentially out of date copy. At any time you can get a clone of the latest pod by calling pod.latest(). If you want to await updates to the manifest, Manifest implements Stream.
The status method now returns an arbitrary type which implements krator::ObjectState, rather than serde_json::Value. This allows you to wrap status-patch logic in an arbitrary type, rather than having to write JSON patches in every state handler.
The async_drop method implemented for PodState has been moved to a method on the newly introduced ObjectState trait, and the AsyncDrop trait has been removed.
We have changed the node label kubernetes.io/os from linux to Provider::ARCH. The reason for this is that a number of vendors appear to use this label as an indication of the types of workloads that can run on the node, and not the host operating system. This is one of the culprits for frequent errors related to DaemonSets like kube-proxy being scheduled on Krustlet nodes. Unfortunately it does not completely eliminate this problem.
Services or Endpoints. This is one of the major focuses of 0.7Our next anticipated version is 0.7.0 (although we will cut a 0.6.1 if necessary). Our main focus for 0.7 will be implementing networking and improving CSI support. During the next release cycle, we will also be moving out the waSCC provider to its own repo. Although we intended for this to occur during this release, we decided it would be better to do after we made these last changes to the provider trait. Full details will be in the 0.7 release notes.
We want to express a huge thanks to all of those in the community who contributed to this release. We appreciate your efforts in making this project a success. As we mentioned before, there were a ton of doc updates from the community and we are very grateful.
Download Krustlet 0.6.0:
Check out our installation docs for information on how to install Krustlet.
Krustlet v0.5.0 is mostly an "under the hood" release if you are just a consumer of Krustlet, although there still are a few useful feature adds! However, if you are using the kubelet crate, there have been significant API changes that lay the groundwork for solid features in the future. These changes are explained below. For more details on what isn't implemented yet, see the Known Issues section.
Because this is pre-release software, there are no backwards compatibility guarantees for the Rust API or functionality. However, we will do our best to document any breaking changes in future releases.
Please note that this is NOT production-ready software, but it is in a usable state. The WASI standard and wasmtime are still under heavy development, and because of this there are key features (like networking) that are missing; these will appear in the future. However, there is networking support available in wasCC.
Provider trait. This includes a new State trait to be use in constructing a state machineSecrets and ConfigMaps into podsImagePullSecrets are now supportedDue to the requirements in Kubernetes 1.19, Krustlet no longer can self label its node object with a role. Full details about why this change was needed can be found here. In practice, this means your node selectors on Pods meant to be run on a Krustlet node should not rely on the role. It is a little unclear what is the best practice here from the Kubernetes standpoint (automation after node joining or some other sort of reconciliation process), but in our experience we find it best to select on the architecture like so:
nodeSelector:
beta.kubernetes.io/os: linux
beta.kubernetes.io/arch: wasm32-wascc
You can see various examples of selectors in our demos.
This section details the major changes to the kubelet crate API. If you do not consume this crate or write your own provider, feel free to skip to the next section.
First and foremost, the Provider trait has been hugely slimmed down. The current trait (with new members commented) as of publishing time is:
pub trait Provider: Sized {
/// The state that is passed between Pod state handlers.
type PodState: 'static + Send + Sync + AsyncDrop;
/// The initial state for Pod state machine.
type InitialState: Default + State<Self::PodState>;
/// The a state to handle early Pod termination.
type TerminatedState: Default + State<Self::PodState>;
const ARCH: &'static str;
async fn node(&self, _builder: &mut Builder) -> anyhow::Result<()> {
Ok(())
}
/// Hook to allow provider to introduced shared state into Pod state.
async fn initialize_pod_state(&self, pod: &Pod) -> anyhow::Result<Self::PodState>;
async fn logs(
&self,
namespace: String,
pod: String,
container: String,
sender: Sender,
) -> anyhow::Result<()>;
async fn exec(&self, _pod: Pod, _command: String) -> anyhow::Result<Vec<String>> {
Err(NotImplementedError.into())
}
async fn env_vars(
container: &Container,
pod: &Pod,
client: &kube::Client,
) -> HashMap<String, String> {
...
}
}
We have removed all of the separate functions for handling the basic CRUD operations as those are now handled by the new state machine (covered below). A few important details about the new parts of the Provider: The InitialState and TerminatedState allow Provider authors to specify what the starting state for a Pod is and the state it should jump to if deleted. The PodState and initialize_pod_state are used in conjunction to allow your Provider to exposed shared state (often stored within the Provider itself) to all of the pods. A simple example of this is the set of currently assigned ports used in the waSCC provider. This shared state must implement the custom AsyncDrop as Rust currently has no async drop standard.
In this new paradigm, providers are expected to provide a full state machine for the lifecycle of the pod. This new design allows for each step of processing to be encapsulated in its own state (e.g. pulling images is one state, starting containers is another). At the heart of this is one new trait, State, and one new enum: Transition as shown below:
pub trait State<PodState>: Sync + Send + 'static + std::fmt::Debug {
/// Provider supplies method to be executed when in this state.
async fn next(
self: Box<Self>,
pod_state: &mut PodState,
pod: &Pod,
) -> anyhow::Result<Transition<PodState>>;
/// Provider supplies JSON status patch to apply when entering this state.
async fn json_status(
&self,
pod_state: &mut PodState,
pod: &Pod,
) -> anyhow::Result<serde_json::Value>;
}
pub enum Transition<PodState> {
/// Transition to new state.
Next(StateHolder<PodState>),
/// This is a terminal node of the state graph.
Complete(anyhow::Result<()>),
}
Each state in your machine must implement the State trait. The json_status function is called when entering into the state to update the status in Kubernetes. The next function is called in an iterative fashion to walk through the state machine. Transitions to other states are done using the Transition next method, which abstracts away some boxing and other things behind the scenes to construct a Transition::Next.
For additional safety, each state must explicitly mark what its next possible states are using the TransitionTo trait, providing compile time guarantees that you aren't transitioning into a state you aren't expecting.
To tie it all together, here is a super simple implementation of a state:
use kubelet::state::{Transition, State, TransitionTo};
use kubelet::pod::Pod;
#[derive(Debug)]
struct TestState;
impl TransitionTo<TestState> for TestState {}
struct PodState;
#[async_trait::async_trait]
impl State<PodState> for TestState {
async fn next(
self: Box<Self>,
_pod_state: &mut PodState,
_pod: &Pod,
) -> anyhow::Result<Transition<PodState>> {
Ok(Transition::next(self, TestState))
}
async fn json_status(
&self,
_pod_state: &mut PodState,
_pod: &Pod,
) -> anyhow::Result<serde_json::Value> {
Ok(serde_json::json!(null))
}
}
If you are interested in the technical details and design decisions behind the state machine, there will be a forthcoming blog post on https://deislabs.io (we will update the link here when it releases). Please note that this API is still needs some polishing work. We don't expect any more major changes like what is in this release, but there will be various ergonomic changes to make it easier to use as we continue to iterate.
For real world examples of a provider implementation, please take a look at the WASI and waSCC providers in Krustlet. See the crate documentation for full documentation and more examples.
Services or EndpointsContainerCreating, CrashLoopBackoff, etc.). However, please note that running and error conditions are supported, so you'll know if your pod is in an erroneous stateOur next anticipated version is 0.6.0 (although we will cut a 0.5.1 if necessary). You can see a full list of issues planned for 0.6 in the milestone. Our main focus for 0.6 will be around supporting cloud provider volumes and continued API ergonomic enhancements. During the next release cycle, we will also be moving out the waSCC provider to its own repo. Full details will be in the 0.6 release notes.
We want to express a huge thanks to all of those in the community who contributed to this release. We appreciate your efforts in making this project a success.
We also would like to call out @kflansburg's herculean efforts with the new state machine API and refactor. He recently joined us as a core maintainer and took the lead on this massive chunk of work. If you see him on the interwebs, please say thanks!
Download Krustlet 0.5.0:
Check out our installation docs for information on how to install Krustlet.
Krustlet v0.4.0 focuses on Windows support, Init Containers, and testing improvements. For more details on what isn't implemented yet, see the Known Issues section.
Because this is pre-release software, there are no backwards compatibility guarantees for the Rust API or functionality. However, we will do our best to document any breaking changes in future releases.
Please note that this is NOT production-ready software, but it is in a usable state. The WASI standard and wasmtime are still under heavy development, and because of this there are key features (like networking) that are missing; these will appear in the future. However, there is networking support available in wasCC.
nodePort isn't specifiedImagePullPolicy is now respected in both providersThe oci-distribution crate has unexported the Client's version and auth methods as they are only meant to be used internally. The Client fetch_manifest_digest method now requires an &mut self.
Services or EndpointsContainerCreating, CrashLoopBackoff, etc.). However, please note that running and error conditions are supported, so you'll know if your pod is in an erroneous stateOur next anticipated version is 0.5.0 (although we will cut a 0.4.1 if necessary). You can see a full list of issues planned for 0.5 in the milestone. We expect 0.5.0 to contain some major breaking changes, particularly around the Provider API. However, we believe these changes will make it easier for developers to write their own providers and simplify the process for updating pod statuses. These changes will be clearly documented in the 0.5 release notes
We also want to express a huge thanks to all of those in the community who contributed to this release. We appreciate your efforts in making this project a success.
Download Krustlet 0.4.0:
Check out our installation docs for information on how to install Krustlet.
Krustlet v0.3.0 is the third release of Krustlet. This release was focused on refactoring, documentation, and TLS bootstrapping. For more details on what isn't implemented yet, see the Known Issues section.
Because this is pre-release software, there are no backwards compatibility guarantees for the Rust API or functionality. However, we will do our best to document any breaking changes in future releases.
Please note that this is NOT production-ready software, but it is in a usable state. The WASI standard and wasmtime are still under heavy development, and because of this there are key features (like networking) that are missing; these will appear in the future. However, there is networking support available in wasCC.
.pfx certificate bundle for TLS in favor of a separate certificate and key. Because of this, the --pfx-password flag has been removed and the new flags for TLS are --cert-file and --private-key-file
KRUSTLET_ prefix:
TLS_PRIVATE_KEY_FILE => KRUSTLET_PRIVATE_KEY_FILE
TLS_CERT_FILE => KRUSTLET_CERT_FILE
kubelet crate have shifted around. When upgrading your Providers to kubelet 0.3.0, note the following changes in your import paths:
kubelet::Provider -> kubelet::provider::Provider
kubelet::NodeBuilder -> kubelet::node::Builder
kubelet::Pod -> kubelet::pod::Pod
kubelet::module_store::ModuleStore -> kubelet::store::Store
kubelet::module_store::FileModuleStore -> kubelet::store::oci::FileStore
kubelet::handle::key_from_pod -> kubelet::pod::key_from_pod
kubelet::handle::pod_key -> kubelet::pod::pod_key
kubelet::handle::PodHandle -> kubelet::pod::Handle
kubelet::handle::RuntimeHandle -> kubelet::container::Handle
kubelet::status::update_pod_status -> kubelet::pod::update_status
kubelet::status::Phase -> kubelet::pod::Phase
kubelet::status::Status -> kubelet::pod::Status
kubelet::status::ContainerStatus -> kubelet::container::Status
kubelet::handle::Stop -> kubelet::handle::StopHandler
kubelet::volumes::VolumeRef -> kubelet::volume::Ref
kubelet::logs::LogOptions -> kubelet::log::Options
kubelet::logs::LogSender -> kubelet::log::Sender
ContainerCreating, CrashLoopBackoff, etc.). However, please note that running and error conditions are supported, so you'll know if your pod is erroringOur next anticipated version is 0.4.0 (although we will cut a 0.3.1 if necessary). You can see a full list of issues planned for 0.4 in the milestone.
We also want to express a huge thanks to all of those in the community who contributed to this release. We had a whole slew of new contributors and their work has been invaluable in improving quality. We appreciate your efforts in making this project a success.
Download Krustlet 0.3.0:
Check out our installation docs for information on how to install Krustlet.
This is purely a maintenance release for the Rust crate. There was a missing feature flag on the docs build that was causing it to fail. Otherwise this has the same functionality as 0.2.0. We apologize for the mixup
Download Krustlet 0.2.1:
Check out our installation docs for information on how to install Krustlet
Krustlet 0.2.0 is the second release of Krustlet. This release was focused on adding features and improving overall ergonomics for running Krustlet. For more details on what isn't implemented yet, see the Known Issues section.
Because this is pre-release software, there are no backwards compatibility guarantees for the Rust API or functionality. However, we will do our best to document any breaking changes in future releases
Please note that this is NOT production-ready software, but it is in a usable state. The WASI standard and wasmtime are still under heavy development, and because of this there are key features (like networking) that are missing; these will appear in the future. However, there is networking support available in wasCC.
All of the functionality of Krustlet is also available as a Rust crate, and this is the first release that we are publishing to crates.io! We have pushed the oci-distribution crate and the kubelet crate for use in other projects. The kubelet crate can be used by anyone who wants to write a Virtual Kubelet style provider in Rust. We'd love to see some other examples of Providers if anyone implements one!
Please remember that the oci-distribution crate is a partial implementation of the OCI spec that we hope to fully flesh out in the future (as of right now, it is just an implementation of what is needed to pull modules for Krustlet)
ContainerCreating, CrashLoopBackoff, etc.). However, please note that running and error conditions are supported, so you'll know if your pod is erroringOur next anticipated version is 0.3.0 (although we will cut a 0.2.1 if necessary). You can see a full list of issues planned for 0.3 in the milestone
We also want to express a huge thanks to all of those in the community who contributed to this release. We had a whole slew of new contributors and their work has been invaluable in improving quality. We appreciate your efforts in making this project a success
Download Krustlet 0.2.0:
Check out our installation docs for information on how to install Krustlet
Krustlet 0.1.0 is the first release of Krustlet. For more details on the project and why it exists, see the blog post. This release contains a minimally functional Kubelet implementation for using WASI ("wasmtime") or wasCC as a backend provider. You can create a pod, have it run (with status updates), and be able to fetch the logs. For more details on what isn't implemented yet, see the Known Issues section.
Because this is pre-release software, there are no backwards compatibility guarantees for the Rust API or functionality. However, we will do our best to document any breaking changes in future releases
Please note that this is not fully production ready software, but it is definitely in a usable state. The WASI standard and wasmtime are still under heavy development. There are some key features (like networking) that are currently missing, but will be made available in future updates. However, there is networking support available in wasCC.
Right now, workloads for the WASI provider should be short lived. There is no current way to safely interrupt a long-running WASM module in wasmtime, though there will be in the future.
All of the functionality of Krustlet is also available as a Rust crate, but since this is still experimental we have not published the crates to https://crates.io. If you are interested in using these as published crates, please let us know so we can prioritize it!
To use any of the crates as a dependency, you'll need to add it to your dependencies like so (replacing kubelet with the name of the crate you want):
kubelet = { git = "https://github.com/deislabs/krustlet", tag = "v0.1.0" }
The kubelet crate is a generic Kubelet implementation that can be used to create your own Kubelet or Provider. The oci-distribution crate is a partial implementation of the OCI spec that we hope to fully flesh out in the future (as of right now, it is just an implementation of what is needed to pull images for Krustlet)
ContainerCreating, CrashLoopBackoff, etc.). However, please note that running and error conditions are supported, so you'll know if your pod is erroringapps/v1 API (e.g. Deployments)Our next anticipated version is 0.2.0 (although we will cut a 0.1.1 if necessary). You can see a full list of issues planned for 0.2 in the milestone
We also want to express a huge thanks to all of those in the community who contributed to this release. We appreciate your efforts in making this project a success
Download Krustlet 0.1.0:
Check out our installation docs for information on how to install Krustlet