June 27, 2022
SPIFFE/SPIRE is a great tool to use when you want to perform workload attestation and mutually authentication in heterogeneous environment. It provides short lived cryptographic identities (called SVIDs) that can be used by workloads to authenticate to other workloads and establish a mutual TLS (mTLS) connection. Currently the SPIRE agent workload API would need to be exposed as a hostPath to the kubernetes cluster such that other workloads could mount the the workload API and request a SVID. hostPath bring a whole slew of drawbacks and security concerns when used in a production environnement. Many hardening guide recommend against using hostPath as it can be a gateway into privilege escalation. In this post, lets see what is the alternative and how it can be used to deploy SPIRE!
SPIFFE (Secure Production Identity Framework for Everyone) is an open-source standard for securely identifying software systems in dynamic and heterogeneous environments. Like I stated above, it can be used to establish mTLS connection or sign and verify JWT tokens. As an open standard, multiple tools implement SPIFFE in some way. These tools are SPIRE (which we will focus on today), Istio (service mesh), HashiCorp Consul (another service mesh) and Kuma (third service mesh).
SPIRE is a production ready implementation of the SPIFEE framework. It perform node and workload attestation in oder to issue SVIDs to workloads. SPIRE consists of a central SPIRE server and SPIRE agents that run on the nodes (one or more depending on the environment). The server acts as the signing authority for all the workloads that are registered. The SPIRE agents are responsible for requesting the signed SVIDs, attesting the identities of the workloads that call the workload API and finally provide the SVID once successfully attested.
The current deployment of SPIRE relies heavily on hostPath. This enables the workload API to be accessible by other workloads and attestation process can occur. Using hostPath in a pod can compromise the security of your cluster and allow an attacker to perform a privilege escalation attack. For example, if the administrator has not limited what can be mounted, an attacker could mount the entire host’s filesystem into your pod. This could give them read and write access on the host’s filesystem! (1)
As hostPath is needed for proper functionality of the SPIRE, is there another way we could deploy SPIRE that is more secure? Yes! Lets talk about that next.
CSI (Container Storage Interface) is an open standard that allows for exposing of block and file storage systems to containerized workloads. Use of the CSI driver allows for the kubernetes volume layer to become extensible. This allows for third-party storage providers to deploy plugins that expose the new storage system in kubernetes without having to touch the core kubernetes code. This results in more storage options for users and makes the system more secure and reliable.
Currently the SPIRE agent is deployed as a DaemonSet where each node has workload API exposed. Each workload would mount the workload API as a hostPath volume. The motivation for the CSI driver creation was to remove the need for the workload pods to mount the workload API. Thus, only the SPIRE agent pod that contains the CSI driver containers would require the hostPath volume mounts (to interact with the Kubelet). This is the only limitation of this driver as using an emptyDir volume would result in the backing directory to be removed if the SPIFFE CSI Driver pod is restarted, invalidating the mount into workload containers.
Note: You can find a script and configs of this example on our github: spiffe-csi-driver
First we need to deploy out the CSIDriver.
These are the important fields (3):
Next we need to create the namespace for spire:
The deployment of the SPIRE server is the same as the normal deployment in k8s. To make things easier, you can deploy using the provided yaml in the following location: https://raw.githubusercontent.com/spiffe/spiffe-csi/main/example/config/spire-server.yaml
The SPIRE server deployment uses the current released version image: ghcr.io/spiffe/spire-server:1.1.1:
The CSI driver integration is all done within the SPIRE agent DaemonSet. Lets break down the DaemonSet to understand which containers are being deployed.
First its the SPIRE Agent container like normal using the image: ghcr.io/spiffe/spire-agent:1.1.1
The next container is the SPIFFEE CSI Driver. Here the important pieces are the volumeMounts which consist of the the SPIRE agent socket, CSI driver socket and the kubelet mount to interact with mounts for containers.
The last container is the CSI Node Driver Registrar which takes care of all the little details required to register a CSI driver with the kubelet.
Finally lets look at the actual volumes that attach to these volumeMounts. hostPath needs to be used here only as the SPIRE agent socket is shared between the containers running in the DaemonSet. The SPIFFE CSI Driver needs to interact with the kubelet hence the path /var/lib/kubelet.
The SPIRE Agent with all the configured CSI driver containers can be easily deployed via this yaml configuration: https://raw.githubusercontent.com/spiffe/spiffe-csi/main/example/config/spire-agent.yaml
Once the SPIRE server and Agent are running in your cluster, the next step it to register. This can be done via:
Here we are using the k8s_psat plugin for the node registration and k8s plugin for the workload. This can be modified based on your needs.
Now that SPIRE server and agent is registered and running, lets test out to make sure that the CSI driver works. We will deploy a client workload that will utilize the CSI driver instead of using the usual hostPath.
Check the workload logs to see the update received over the Workload API:
The output should be similar to:
If you are having trouble getting things to run, you can visit our github to find the full example and script to get it running quickly: spiffe-csi-driver
As I have described in this post, using the hostPath can be a very big security risk. With the traditional SPIRE deployment, each workload pod also needed the hostPath mount attached to access the Workload API. With the CSI Driver, this limits the use of hostPath to only the SPIRE agent deployment. This still gives you all the benefits of using SPIRE in your environment without having to use hostPath.
The latest industry news, interviews, technologies, and resources.