There's a common pattern for on-premises clusters: how to securely access cloud host resources? This requirement can play out in multiple ways:

• Route 53 - You want to use Let's Encrypt for certificates and want to validate domain ownership via the automated Route53 provider
• S3 - Your applications want to store object data or even just have off-site backups
• Hosted LLMs - Enterprise plans are starting to support local identity providers instead of static API keys

This list can extend to databases, logging services, and really any kind of cloud service you may want to consume.

When connecting to these services, it's typical to get a static API key that needs to get rotated. In the zero-trust world, this approach isn't preferred and can lead to some real issues. Since the keys are long lived they become difficult to track and even harder to determine if they've been lost due to a vulnerability or a bug. This has led to the creation and deployment of API proxies to manage these static keys, which can be a security hassle its self!

The better approach is to use your workload's own identity to authenticate to these services. How does workload identity work in Kubernetes and how can you leverage that identity for a zero trust infrastructure? We'll walk through how Kubernetes assigns identities to each workload and how your services can use those identity, and some of the pitfalls when using Kubernetes identities directly with external services. Then we'll look at how using OpenUnison's Security Token Service help avoid those issues.

Supporting Cloud Hosted Services

Before we dive into how Kubernetes workload identity has evolved and OpenUnison's STS, it's important to look at how an on-prem system using a cloud hosted service is different then working all in the cloud or all on-prem. Let's use the example of a GitHub action making a call to AWS services.

Your Github action requests a token for the audience sts.aws.com, and you configure AWS to trust github.com as an issuer. You're then able to present your tokens to AWS when using APIs and get AWS credentials to use with your APIs. Since both AWS and GitHub are on the public cloud, this works great. What if your cluster is on-premesis or running on a private subnet?

If your identity provider isn't hosted on the internet, you tell your identity provider to issue tokens from a URL that is public, then post the OIDC discovery documents there. That way the token can be validated by the cloud service without having direct access to your identity provider.

Now that we've seen how you would format your tokens for validation by a cloud based service, let's next look at how Kubernetes' workload identity can be used to get tokens for these cloud hosted services.

Kubernetes Workload Identity

So far, we've looked at how on-premesis workloads can use cloud hosted services with local identity providers. Now we'll look at how Kubernetes handles workload identity for services outside of the cluster.

Since version v0.4 Kubernetes has mounted ServiceAccount tokens into Pods to support providing each workload an identity it could use to communicate with the API server. These tokens didn't have an expiration, and were stored as Secrets. You could, in theory, use these tokens as an identity to an external service, but you would need to manually verify them against a public key. It was also dangerous to use these tokens for external services, because if the external service were to abuse this token, they now have an identity to use against your API server!

In 1.24, the TokenRequest API went GA and some very important changes happened in Kubernetes:

• Projected Tokens - Instead of tokens being stored as Secrets, they were "projected" into containers as volumes, meaning that tokens didn't need to be stored as Secrets anymore.
• Token Expiration - ServiceAccount tokens associated with Pods now had expirations (more on that later), so they couldn't be used forever.
• Workload Identifier - Each token is bound to a ServiceAccount AND a specific workload. This means even though a token may not have expired, it's important to make sure that the workload it's tied to is still running.
• Accessible Issuer URL - You can make the OIDC discovery information publicly queryable so external services can validate that tokens have not expired.

With 1.24, you could configure an external service, like Vault, to trust your API server's discovery document and then let your Pod get secrets using that identity! There are some traps to avoid though:

• Easy to use default tokens - When creating a token bound for a remote service, you're supposed to create a second token scoped for that service. Unfortunately, many services make it easy to configure any audience you want so there's no way to enforce this rule. It's important to create a service specific token because you don't want to send a token outside the cluster that can be used against the API server.
• Single key for all services - It doesn't matter which service you're integrating with, there's only one key in your cluster for signing ServiceAccount tokens. If you want to rotate it, you need to rotate the same key used by Pods to talk to the API server. You also can't segragate services by using different keys, a common security practice to protect against losing keys.

You can customize the issuer for Kubernetes ServiceAccount tokens. By default, it's often the URL of the API server for cloud hosted clusters and just https://kubernetes.svc.default for on-premesis clusters. The downside to this customization is that it requires a restart of your API servers and is a cluster-wide change.

Finally, using a ServiceAccount token with remote services loses out on one of these tokens' best security features: validating that a token is associated with a running workload. When you send a token to a cloud hosted service, you can't validate that it's still associated with a running workload since your service can't make an API call back to your cluster. Most distrobutions' default token length is one year. If you're generating custom tokens for your services, you can bring that down to ten minutes but it would require additional controllers to enforce in your environment.

While Kubernetes has made fantastic strides to expand it's security posture with respect to identity, it's not enough to enable a hybrid environment. A Security Token Service will give you the flexibility to leverage these identity capabilities to connect to any service, whether in the cloud or on-premesis.

Using OpenUnison's Security Token Service with Kubernetes

While you may have used OpenUnison's authentication service before, OpenUnison also supports security token services. What's unique about OpenUnison, especially in the Kubernetes world, is we're Kubernetes native. Our STS doesn't just work on validating JSON Web Tokens (JWTs). OpenUnison knows how to validate that the tokens are still associated with a workload. Another major benefit to OpenUnison is our built in capability to be a mutating webhook and admission controller, so OpenUnison can update your workloads automatically with the correct configuration without needing to implement custom logic. Let's look at how the OpenUnison STS integrates your workloads with external services:

1. When OpenUnison is configured as an STS it generates the appropriate discovery documents for the configured issuer. These documents are then uploaded to a public system like AWS CloudFront via an S3 bucket. This process can be automated to make key rotation easier.
2. Once a Pod is created, if it has a specific label, it is submitted to the OpenUnison mutating webhook which injects with a small sidecar to generate and keep a token updated. This sidecar is a scratch container with a single binary in it.
3. As the sidecar runs, it takes the Pod's ServiceAccount token and requests a token for the remote service from OpenUnison.
4. OpenUnison validates the ServiceAccount token by making sure its associated with a running workload.
5. The Pod uses the token from OpenUnison with the external service, which validates it against the publicly hosted OIDC discovery documents.

The process of using the sidecar is all automated, from the deployment to its invocation. The token generated is available to the main pod and the external service now has a unique token that is short lived. Also, you have the freedom to create and define multiple keys for different services and rotate them as regularly as you want. Finally, your Pod's ServiceAccount token never leaves your cluster.

Want to give the OpenUnison STS a try? We've published step-by-step instructions for working with AWS and Vault. If you're interested in another service, let us know by opening an issue on GitHub or by reaching out to our sales team!