Skip to main content

3 posts tagged with "gardener-extensions"

View All Tags

· 2 min read
Jens Schneider

Thanks to everyone

In the last week of September 2022, I participated in the Gardener Hackathon. Unfortunately, I need to attend the event remotely, as I caught Covid the week before the event. Therefore, I want to thank everyone making this experience possible for me, anyway.

Even remote hacking together is great

As Tim also participated remotely, we formed a remote hacking team contributing to the development of the Gardener extension-registry-cache. Of course, we were in touch with the on-site contributors via video calls which laid the foundation for three highly productive hacking days. Right after tying down a common todo list, we distributed the workload to the on-site and remote teams, and started hacking. From time to time, we held synchronization meetings, so that everyone was up-to-date and the current state of work was not only reflected in commits, branches, and pull request but also communicated to everyone in the team.

Beyond the internal communication within the team, Tim and Rafael organized a demo session on day 2. It was really amazing to see the progress made with respect to the various topics covered by the hackathon.

Conclusion

Clearly, the hackathon was a great event, and even hacking together remotely was a great experience. Of course, the social aspect of working together on-site cannot be mimicked. Therefore, I am already looking forward to the next Gardener hackathon which I hopefully can attend on-site.

· 12 min read
Jens Schneider

TLDR;

Recently, we developed the gardener-extension-mwe, which serves as a minimal working example for Gardener extensions. If you are only interested in the code, go and checkout the repository on github. If you want to learn more, keep on reading.

Table of Contents

Introduction

Starting the development of new Gardener extensions can be challenging. As the Gardener documentation is fairly complex and driven by the history of the project, getting into the overall concepts is not easy. Moreover, code examples for Gardener extensions reside in separate Git repositories lacking documentation. However, early in March 2022 the gardener-extension-shoot-networking-filter was published, which comes at a more beginner friendly level than the e.g. the cloud-provider extensions. In particular, it extends Shoot clusters by the use of managed resources, which might be more straight-forward than the interaction with a cloud service provider as performed by e.g. the gardener-extension-provider-aws. Thus, gardener-extension-shoot-networking-filter provides a reasonable starting point for new developments, which target at automated deployments into Shoot clusters.

However, going beyond the identification of a starting point, it makes sense to take a closer look at the concepts for extension development. In the extension directory of the Gardener Git repository, we find several Go-packages defining interfaces, which can be implemented by extension controllers. Put simply, we can search for files matching pkg/controller/*/actuator.go, in order to find interfaces for controllers acting on the corresponding resources. E.g., if our controller defines a (golang)-type -- called actuator -- implementing the interface defined in pkg/controller/extension/actuator.go, the controller will reconcile resources of type *extensionsv1alpha1.Extension. Consequently, the controller will take care for all the steps we define in the Reconcile method of the actuator, when Extension resources are updated. Of course, there might be more complex scenarios where reconciling Extension resources would be insufficient. In these cases, other interfaces like e.g. defined in pkg/controller/infrastructure/actuator.go would need to be implemented. However, these cases lie beyond the scope of this blog post.

Next, we will dive into some basic workflows for Gardener extension development.

Basic workflows

In software engineering, it is reasonable to develop on a local machine with a controllable toolchain. As already mentioned above, Gardener extensions are implemented in Go. Therefore, let's identify a few requirements for the development:

  • An installation of Go
  • A text editor, which (optionally) supports gopls
  • A Go debugger, which is most likely to be Delve
  • A Gardener development environment. This can be setup by
    • Running Gardener locally (also checkout #5548, if you are running on Linux)
    • Setting up a development Gardener on some cloud infrastructure. This definitely comes closer to the real world scenario your extension will eventually live in. The block diagram below depicts the overall setup including the requirements from above.
┌──────────────────────────────────────────────┐
│ Development Computer │
├──────────────────────────────────────────────┤
│ │
│ ┌──────────────────────────────────────────┐ │
│ │ - Your toolchain │ │
│ └──────────────────────────────────────────┘ │
│ │
│ ┌────────────┐ ┌──────────────┐ │
│ │Kubeconfigs │ │Your code │ │
│ └──┬──────┬──┘ └────────┬─────┘ │
│ │ │ │ │
└────────┼──────┼────────────────────┼─────────┘
│ │ │
│ │apply │
apply │ │resources │reconcile
resources│ │ │resources
│ └──────────────────┐ │
│ │ │
│ │ │
▼ ▼ ▼
┌─────────────────┐ ┌─────────────────┐
│ Garden Cluster │ │ Seed Cluster │
├─────────────────┤ ├─────────────────┤
│- Project │ │- Extension │
│- Seed │ │- Controller │
│- Shoot │ │- ... │
│- ... │ │ │
└─────────────────┘ └─────────────────┘

As you can see, the code for the extension controller is running on your local development machine, and reconciles resources, such as Extension resources, in the Seed cluster. Of course, you will not have dedicated clusters for the Garden cluster and Seed cluster, when running Gardener locally. However, the overall buidling blocks stay conceptually the same. Once these requirements are met, you are good to go for your first steps with Gardener extensions. Wait! I have to setup an entire Gardener, if I want to rapidly prototype an extension controller? Yes and No. Depending on your development/test case, it might be reasonable to "fake" a Gardener environment on a vanilla Kubernetes cluster. We will get to this development case below. For rock solid testing, however, you most probably need a real world Gardener environment.

The mininal working example

As of May 2022, we provide a Minimal Working Example (MWE) for gardener extensions. Of course, this example did not come out of nowhere. Therefore, we review the development process and break the example down to its components in the following: Taking a look at other extensions, it is observed that we need some boiler plate code for running our controller, so that it works together with all the other Gardener components. For the MWE, we collected the relevant files and adjusted them to our needs. Thus, we can have a look at the cmd directory of the gardener-extension-mwe and find a simple structure with 3 files, which are responsible for starting our controller and ensuring that it acts on the defined resources.

cmd
└── gardener-extension-mwe
├── app
│   ├── app.go
│   └── options.go
└── main.go

If you want to start the development of a new extension, you do not need to worry too much about these files. Most probably, you can simple copy them over, and adjust some variables to your needs. Actually, we also copied these files from the gardener-extension-shoot-networking-filter and adjusted them to the needs of the MWE. Given that we have the boilerplate code in the cmd directory available now, we can go ahead and define a type which implements an actuator interface. For this, we need the files in the pkg directory. Lets take a look at the structure:

pkg
└── controller
└── lifecycle
├── actuator.go
└── add.go

Also here, we find only two files, and the implementation of the interface is located in actuator.go. This is the place where most of the magic of your new controller happens. In the case of the MWE, the actuator will only output logs, when Extension resources are reconciled. Obviously, all code is written in Go and consequently we will also need to pull in some dependencies. For this, we need the files go.mod and go.sum. Typically, the source code of the dependencies is also committed to the repository, which comes at advantages and downsides. The main advantage is that all code needed for building an application is available in the repository. On the other hand, committing several 1000s lines of code during vendoring clutters the commit history of the repository. Therefore, we only provide the files mentioned above in the initial commit of the MWE and perform the vendoring (by running go mod vendor) in another commit. At this state of the repository, we can already start and go the first steps with our new controller in a vanilla Kubernetes cluster.

Rapid prototyping on a Kubernetes cluster (tested with version 1.22.6)

Assuming you have read the basic workflows section, we are ready to dive into the exemplary development techniques. So let's fetch the code and setup the repository:

git clone https://github.com/23technologies/gardener-extension-mwe.git
cd gardener-extension-mwe
git checkout 3c238bd # checkout the commit containing first vendoring
mkdir dev
cp PATH-TO/KUBECONFIG.yaml dev/kubeconfig.yaml

Now, we can already start our controller and should get some output showing that it was started:

go run ./cmd/gardener-extension-mwe --kubeconfig=dev/kubeconfig.yaml  --ignore-operation-annotation=true --leader-election=false

However, we will not observe any other output, since the controller is still freewheeling. Remember, reconciliation will be triggered on Extension resources. As our vanilla Kubernetes cluster does not know anything about Extension resources yet, we will have to "fake" the Gardener environment. In other Gardener extensions, we find resources for a "fake" Gardener setup in the example directory. Therefore, we prepared the example directory in another commit. Let's check it out: Open a new terminal pane and navigate to your repository and go for

git checkout 50f7136 # this commit entails the example directory
export KUBECONFIG=dev/kubeconfig.yaml
kubectl apply -f example/10-fake-shoot-controlplane.yaml
kubectl apply -f example/20-crd-cluster.yaml
kubectl apply -f example/20-crd-extension.yaml
kubectl apply -f example/30-cluster.yaml

Now, the cluster simulates a Gardener environment and we can apply an Extension resource:

kubectl apply -f example/40-extension.yaml

Take another look at the terminal running our controller now. It should have logged a "Hello World" message. Of course, we can also delete the Extension resource again and the controller will tell us that the Delete method was called.

kubectl delete -f example/40-extension.yaml

As we have the code and a method to trigger its execution available now, we can go ahead for a more interactive approach based on the Delve debugger. Let's start all over again and run our controller using Delve

dlv debug ./cmd/gardener-extension-mwe -- --kubeconfig=dev/kubeconfig.yaml  --ignore-operation-annotation=true --leader-election=false

and we will end up in a commandline with a (dlv) prompt. Next, we ask dlv to break in the Reconcile method

(dlv) b github.com/23technologies/gardener-extension-mwe/pkg/controller/lifecycle.(*actuator).Reconcile

and continue the execution of the controller

(dlv) c

Afterwards, you should observe some output of the controller, again. However, Delve will not break the execution until the Reconcile method is called. Thus, we apply the Extension resource once again

kubectl apply -f example/40-extension.yaml

and Delve will stop in the Reconcile method. Now, you can step through the code, see where it enters code paths pointing into the vendor directory, and inspect the values of certain variables. Obviously, the amount of variables you can inspect is limited in the MWE, but e.g. we can have a look at the *extensionsv1alpha1.Extension passed to the Reconcile method

(dlv) p ex.ObjectMeta.Name

which should print "mwe". Generally, this is a great way to approach unknown software, since you will quickly get a feeling for the different components. Thus, we expect that you can benefit from this workflow, when developing your own extensions. Even though this approach offers capabilities for rapid prototyping, it is still limited, since we cannot act e.g. on Shoot clusters as available in a real world Gardener. Therefore, we step into the development in a Gardener environment in the next section.

Development in a real Gardener environment

Developing and testing our extension in a real world Gardener requires a ControllerRegistration resource in the Garden cluster causing the installation of the controller in Seed clusters. Generally, the installation performed by Helm charts and consequently, we need to provide these charts in the repository. Also for the MWE, we prepared the charts directory containing all relevant Helmcharts for the deployment of the controller. Note that this set of charts is very limited and in production scenarios you might want to add something like a VerticalPodAutoscaler as done e.g. in the gardener-extension-shoot-networking-filter. However, implementing production ready charts goes beyond the scope of this post, and consequently the MWE charts were added in another commit. These charts target at running the controller in Seed clusters. Thus, in charts/gardener-extension-mwe/values.yaml, the image for the deployment is defined. However, we do not want to push that image to a public container registry for each and every change we make to our code. Moreover, we want to run the controller on our local machine for development purposes. Therefore, we need to tweak the values before generating the controller-registration.yaml. Let's go through it step by step:

git clone https://github.com/23technologies/gardener-extension-mwe.git
cd gardener-extension-mwe
mkdir dev
cp PATH-TO/KUBECONFIG-FOR-SEED.yaml dev/kubeconfig.yaml

Next, we generate the controller-registration.yaml, such that the controller is not deployed to the seed cluster and we can hook-in the controller running locally. In particular, we set replicaCount=0 and ignoreResources=true in ./charts/gardener-extension-mwe/values.yaml, before generating the controller-registration.yaml:

yq eval -i '.replicaCount=0 | .ignoreResources=true' charts/gardener-extension-mwe/values.yaml
./vendor/github.com/gardener/gardener/hack/generate-controller-registration.sh mwe charts/gardener-extension-mwe v0.0.1 example/controller-registration.yaml Extension:mwe

Now, let's deploy the generated controller-registration.yaml into the Garden cluster:

export KUBECONFIG=PATH-TO/GARDEN-CLUSTER-KUBECONFIG.yaml
kubectl apply -f example/controller-registration.yaml

From now on, Extension resources of the type mwe will be deployed to Seed clusters when new Shoot clusters with

---
apiVersion: core.gardener.cloud/v1beta1
kind: Shoot
metadata:
name: bar
namespace: garden-foo
spec:
extensions:
- type: mwe
...

are created. In our controller, the Reconcile method will be triggered, when these Extension resources are reconciled. Therefore, we can run the extension controller with Delve now

dlv debug ./cmd/gardener-extension-mwe -- --kubeconfig=dev/kubeconfig.yaml  --ignore-operation-annotation=true --leader-election=false --gardener-version="v1.44.4"

and we can perform debugging operations as explained above. Remember, Delve will not break the execution until the Reconcile method is called. Now, Gardener will create Extension resources for Shoots which will trigger the Reconcile method of our controller. Consequently, we need a new terminal pane in the repository root and execute

export KUBECONFIG=PATH-TO/GARDEN-CLUSTER-KUBECONFIG.yaml
kubectl apply -f example/50-shoot.yaml

Note that it will take some time until the corresponding Extension resource will be created in the Seed cluster. Hang on tight and wait for the Reconcile method being executed. You can start investigating where your code goes using Delve now. Happy hacking!

Last words

This blog post shares our experience, when getting started with Gardener extension development. We hope that this contribution helps you get started more quickly than us. If you have any comment or ideas for improvements, do not hesitate to contact us. We are always willing to improve our work.

· 7 min read
Jens Schneider

TLDR;

Recently, we developed the gardener-extension-shoot-flux, which enables preconfiguring Shoot clusters. If you want to give it a try, go and checkout the repository on Github. If you want to learn more, keep on reading.

Table of Contents

Introduction

Flux offers a set of controllers allowing for reconciling a Kubernetes cluster with a declarative state defined in e.g. a Git repository. Thus it enables GitOps workflows for Kubernetes clusters. Moreover, it provides a general approach of deploying software components into Kubernetes clusters. Gardener is a multi cloud managed Kubernetes service allowing end users to create clusters with a few clicks in its dashboard. However, the user will obtain a vanilla Kubernetes cluster and has to take care for all the components to be deployed into it. Of course, the deployment can be performed manually by applying Kubernetes manifests to the cluster. On the other hand, tools like Flux can help to keep track of the deployments and automate the overall process. Thus, the combination of Gardener and Flux features the potential of creating new Kubernetes clusters in a pre-defined state. For the end users, this results in the seamless creation of clusters with all components on their wish list installed. The gardener-extension-shoot-flux bridges the gap between Gardener and Flux and allows for reconciliation of Shoot clusters to resources defined in a Git repository. By concept, the extension operates on a per-project basis so that clusters in different projects can be reconciled to different repositories.

The rest of this post is organized as follows: First, we will review a few use cases for this extension. Further, the general concept of the extension is outlined, and finally we provide an example on how to use the extension.

Example use cases

Development

Imagine you are developing software which will eventually run on a Kubernetes cluster in the public cloud. Moreover, you and your colleagues want to be able to perform some end-to-end tests besides running your local test suite. For these end-to-end test, an environment mimicking the final production environment is required. Therefore, you might need tools like cert-manager or MinIO. However, you do not want keep several testing clusters in the public cloud available for economic reasons and, in consequence, you need to create new clusters on demand. In this case, the gardener-extension-shoot-flux comes handy, since it allows to configure the cluster asynchronously. Put simply, you can define the desired state of your cluster in a Git repository, and the new clusters will be reconciled to this state automatically. Eventually, this will save the effort to configure the clusters each and every time manually. Of course, you could achieve something similar by hibernation of the development clusters. However, in that case you are less flexible, since throwing away the cluster in case you lost track of your clusters state comes at the price of reconfiguring the entire cluster.

CI/CD

Similar to the development use case above, you might want to run your CI/CD pipeline in Kubernetes clusters coming with a few components already installed. As your pipeline runs frequently, you want to create clusters on the fly or maybe pre-spawn just a few of them. In order to keep your pipeline simple, you can use the gardener-extension-shoot-flux for the configuration of your CI/CD clusters. This way your pipeline can focus on the actual action and does not have to perform the cluster configuration beforehand. This most probably results in cleaner and more stable CI/CD pipelines.

General concept

The general concept of this extension is visualized in the block diagram below.

                 ┌─────────────────────────────────────────────────────────┐
│ Gardener operator │
├─────────────────────────────────────────────────────────┤
│ - A human being │
│ ├────────────┐
│ │ │
│ │ │
└────────┬────────────────────────────────────────────────┘ │
│ ▲ │configures
│deploys │ │SSH-key
│Configmap │read SSH-key │
│ │ │
▼ │ │
┌────────────────────────────────────┴───────────────────┐ │
│ Garden cluster │ │
├────────────────────────┬─────────────────────────┬─────┤ │
│ Projetct 1 │ Project 2 │ ... │ ▼
├────────────────────────┼─────────────────────────┼─────┤ ┌─────────────────────┐
│- Configmap containing │- Configmap containing │ │ │ Git repository │
│ flux configuration │ flux configuration │ │ ├─────────────────────┤
│ │ │ │ │ - Configuration for │
┌───►│- ControllerRegistration│- ControllerRegistration │ ... │ │ shoot clusters │
│ │ │ │ │ └─────────────────────┘
│ │- Shoot with extension │- Shoot with extension │ │ ▲
│ │ enabled │ enabled │ │ │
│ │ │ │ │ │
read config │ │ │ │ │ │
and generate│ └────────────────────────┴─────────────────────────┴─────┘ │reconcile
SSH-keys │ │
│ ┌────────────────────────┐ ┌────────────────────────┐ │
│ │ Seed cluster │ │ Shoot cluster │ │
│ ├────────────────────────┤ ├────────────────────────┤ │
│ │- Controller watching │ │ │ │
└────┼─ extension resource │ │- Flux controllers ────┼──────────────┘
│ │ │ │ │
│ │deploys │ │- GitRepository resource│
│ │ │ │ │
│ ▼ │ │- A main kustomization │
│- Managed resources │ │ │
│ for flux controllers │ │ │
│ and flux config │ │ │
│ │ │ │
└────────────────────────┘ └────────────────────────┘

As depicted, the Gardener operator needs to deploy a ConfigMap into the Garden cluster. This ConfigMap holds some configuration parameters for the extension controller. Moreover, the Gardener operator needs to configure an SSH-key for the Git repository in case of a private repository. This key can be read from the Secret called flux-source in the Garden cluster which is created by the extension controller. Of course, the process of adding the SSH-key to the repository depends on the repository host. E.g. for repositories hosted on Github, the key can simply be added as "Deploy key" in the web-interface.

The extension controller is running in Seed clusters. Besides generating Secrets containing SSH-keys, it reads the configuration from the Garden cluster and creates Managedresources to be processed by the Gardener Resource Manager. These Managedresources entail the resources for the Flux controllers, a GitRepository resource matching the configuration, and a main Kustomization resource. Once the Gardener Resource Manager has deployed these resources to the Shoot cluster, the Flux controllers will reconcile the cluster to the state defined in the Git repository.

You might wonder how the communication between Seed clusters and Garden cluster is established. This is achieved by making use of the Secret containing the gardenlet-kubeconfig which should be available, when the gardenlet is run inside the Seed cluster. Most probably, this is not the most elegant solution, but it resulted in a quick first working solution.

Example Usage

Of course, you need to install the extension before you can use it. You can find ControllerRegistrations on our Github release page. So, you can simply go for

export KUBECONFIG=KUBECONFIG-FOR-GARDEN-CLUSTER
kubectl -f https://github.com/23technologies/gardener-extension-shoot-flux/releases/download/v0.1.2/controller-registration.yaml

in order to install the extension.

For an exemplary use of the extension, we prepared a public repository containing manifest for the installation of Podinfo. As a Gardener operator you can apply the following ConfigMap to your Garden cluster

apiVersion: v1
kind: ConfigMap
metadata:
name: flux-config
namespace: YOUR-PROJECT-NAMESPACE
data:
fluxVersion: v0.29.5 # optional, if not defined the latest release will be used
repositoryUrl: https://github.com/23technologies/shootflux.git
repositoryBranch: main
repositoryType: public

As the repository is public you can create a new Shoot now and enable the extension for this Shoot. Take the snipped below as an example.

apiVersion: core.gardener.cloud/v1beta1
kind: Shoot
metadata:
name: bar
namespace: garden-foo
spec:
extensions:
- type: shoot-flux
...

Gardener will take care for the Shoot creation process. As soon as you can, you can fetch the kubeconfig.yaml for your new Shoot from e.g. the Gardener dashboard. Now, you can watch this cluster by

export KUBECONFIG=KUBECONFIG-FOR-SHOOT
k9s

and you should see that a podinfo deployment should come up. Great! You successfully created a Shoot with the gardener-extension-shoot-flux.