Introducing Multicluster-Controller

A Library for Building Hybrid and Multicloud Kubernetes Operators

TL;DR: https://github.com/admiraltyio/multicluster-controller

Hybrid and multicloud architectures are becoming prevalent, either as a strategy or simply as a result of history and/or mergers and acquisitions. Luckily, to help reduce the inherent complexity, Kubernetes is standardizing the way clouds are operated: the same workflow can be used to manage resources in any cloud, whether public or private. However, managing workloads across clouds is still a challenge. Technically, you could create a single Kubernetes cluster encompassing your entire infrastructure, but that could invalidate some of the assumptions made in the design of Kubernetes itself. Also, you would miss out on turn-key Kubernetes distributions. A more common approach is to operate multiple clusters.

Clusters are isolated from each other by default, which helps with:

• fault isolation;
• trust boundaries;
• only paying for a top-tier service level in production;
• enforcing geographical regulations;
• etc.

However, cluster boundaries get in the way when you'd like to manage the following globally:

• scheduling and autoscaling (ensuring high availability and low latency at the lowest cost);
• service discovery;
• storage;
• monitoring;
• backups and migrations;
• policy enforcement;
• etc.

We need tools to manage resources across clusters. Specific solutions exist. Notably, federation-v2 can sync workloads and route traffic across clusters. To do so, it uses the concepts of Templates, Placements, and Overrides, propagating resources with a push reconciler.

While building a multicluster scheduler at Admiralty (stay tuned), we needed a lower-level abstraction: namely the controller pattern (sometimes called the operator pattern), but for resources in multiple clusters. We needed a tool like the Operator SDK or Kubebuilder (see comparison in a previous blog post), but supporting multiple clusters. Unfortunately, their designs don't allow that. Their APIs would have to change significantly. So, rather than submit a pull request, we decided to make our own tool. Luckily, we were able to leverage parts of controller-runtime, the library powering Kubebuilder and now also the Operator SDK.

Today, we're open-sourcing multicluster-controller. Check out the README for more details on how it works, including how it can be used with custom resources (using CRDs). We've also included a few examples. We hope that the community will find the project useful. (Anyone volunteering to build a multicluster Prometheus operator?)

Warning: though we're already using multicluster-controller internally with great success, the project is still in its infancy and the API may break in future releases. Also, a few must-have features are still in the works, like cross-cluster garbage collection.

Contributions, feature requests and bug reports are welcome. Also, please contact us if you have any questions regarding multicluster-controller or Admiralty.

Acknowledgements#

Many thanks to all the Kubernetes authors, especially those of controller-runtime, apimachinery, and client-go.