One of the nags many people get from running Kubernetes CIS security benchmarks is not using proper CA validation for kubelet serving certificates.

Rule 1.1.21: Verify kubelet's certificate before establishing connection.

Why is that important? By default, each kubelet will generate its own serving certificate as a self-signed certificate. The nagging is good as using self-signed certificates makes it easy to have MITM type of breaches.

Enabling Automatic Certificate Bootstrapping

So luckily this is something that is kinda designed into Kubernetes system components. So we can enable serverTLSBootstrap: true in kubelet config file and everything will fall in place automagically. Almost, but no cigar. From kubelet point of view this is all we need as it will make kubelet to create a proper CSR waiting to be approved and once approved it will take the properly signed certificate and use it as the serving certificate. But the problem is that there's nobody doing the approving automatically. :(

Currently Kubernetes controller manager does NOT approve kubelet serving certificate CSRs automatically. This is mentioned in the documentation:

To use RotateKubeletServerCertificate operators need to run a custom approving controller, or manually approve the serving certificate requests.

Manually approve? No thanks. So we need some "custom approving controller".

Rubber Stamper to the Rescue

As we, as in all users of Pharos, want to have capability to run Pharos in dynamic environments so there needs to be some controller that automatically approves the certificate requests. This is very important e.g. for environments where nodes can be dynamically created by autoscaling or other such functionality.

So we've created kubelet-rubber-stamp as a fully automated kubelet serving CSR approver. The process looks something like this:

kubelet csr approval process

The validation has few of steps:

  • Recognize that the CSR request is for kubelet serving cert
  • CSR has been created to adhere to kubelet certificate "rules"
    • has proper addresses
    • has proper x509 usage policies defined
    • has proper names for both org and common name
  • SubjectAccessReview passes, the given node/kubelet has proper RBAC rules bound.

These checks are actually pretty similar checks what the out-of-box kubelet client certificate approver does.

In a cluster with kubelet-rubber-stamper active, this is what happens during the cluster bootstrap:

$ kubectl get csr
NAME        AGE   REQUESTOR                          CONDITION
csr-5rg65   10m   system:node:demo-pharos-worker-0   Approved,Issued
csr-7lh9z   10m   system:node:demo-pharos-master-0   Approved,Issued
csr-qt86h   10m   system:node:demo-pharos-worker-2   Approved,Issued
csr-wrh4x   10m   system:node:demo-pharos-worker-1   Approved,Issued

All the nodes (actually kubelet on the node) has requested a certificate and rubber-stamper has auto-approved those. Neat.

Custom Controller, lot of Work?

In all honesty, much of the "heavy-lifting" in our custom CSR approving controller is actually managed by the fantastic operator-sdk framework. It makes creating these kind of custom controllers really straight-forward. In this case we only need to wire it up so that we watch CertificateSigningRequest objects in certificates.k8s.io/v1beta1 and act upon new CSRs.

As this operator is Golang based we just bake it into a slim image and deploy it onto a cluster. One thing to note of course is that the deployment needs to create also proper RBAC rules so that the rubber-stamper has enough privileges to actually approve the CSRs.

Pharos FTW

Starting from Pharos 2.2.0 kubelet-rubber-stamp is integrated into the Kubernetes bootstrapping and configuration process. So all your kubelets now come with fully automated certificate bootstrapping for both client and serving certificates.