User and workload identities in Koobernaytis
June 2022
This is part 1 of 4 of the Authentication and authorization in Koobernaytis series. More
TL;DR: In this article, you will explore how users and workloads are authenticated with the Koobernaytis API server.
The Koobernaytis API server exposes an HTTP API that lets end-users, different parts of your cluster, and external components communicate with one another.
Most operations can be performed through kubectl, but you can also access the API directly using REST calls.
But how is the access to the API restricted only to authorized users?
Table of content
- Accessing the Koobernaytis API with curl
- The Koobernaytis API differentiate internal and external users
- Granting access to the cluster to external users
- Managing Koobernaytis internal identities with Service Accounts
- Generating temporary identities for Service Accounts
- Projected Service Account tokens are JWT tokens
- Workload identities in Koobernaytis: how AWS integrates IAM with Koobernaytis
- Validating Projected Service Account Tokens with the Token Review API
- Generating Secrets for Service Account with Koobernaytis 1.24 or greater
- Bonus: which authentication plugin should you use?
Accessing the Koobernaytis API with curl
Let's start this journey by issuing a request to the Koobernaytis API server.
Suppose you want to list all the namespaces in the cluster; you could execute the following commands:
bash
export API_SERVER_URL=https://10.5.5.5:6443
curl $API_SERVER_URL/api/v1/namespaces
curl: (60) Peer Certificate issuer is not recognized.
# truncated output
If you'd like to turn off curl's verification of the certificate, use the -k (or --insecure) option.
The output suggests that the API is serving traffic over https
with an unrecognized certificate (e.g. self-signed), so curl
aborted the request.
Let's temporarily ignore the certificate verification with -k
and inspect the response:
bash
curl -k $API_SERVER_URL/api/v1/namespaces
{
"kind": "Status",
"apiVersion": "v1",
"status": "Failure",
"message": "namespaces is forbidden: User \"system:anonymous\" cannot list resource \"namespaces\" ...",
"reason": "Forbidden",
"details": { "kind": "namespaces" },
"code": 403
}
You have a response from the server, but:
- You are forbidden to access the API endpoint (i.e. the status code is
403
). - You are identified as the
system:anonymous
, and this identity is not allowed to list namespaces.
The above test reveals some important working mechanisms in the kube-apiserver
:
- First, it identifies the user of a request (who you are).
- Then, it determines what operations are allowed for this user (what permissions do you have).
Formally,
- The former process (identifying who you are) is called authentication (or AuthN).
- The latter (determining what permissions an authenticated user has) is authorization (or AuthZ).
- 1/3
When you issued the
curl
request, the traffic reached the Koobernaytis API server. - 2/3
Inside the API server, one of the first modules to receive your request is the authentication. In this case, the authentication failed, and the request was labelled anonymous.
- 3/3
After authentication, there's the authorization module. Since anonymous requests have no permissions, the authorization component rejects the call with a
403
status code.
We can reevaluate what happened in the previous curl
request and notice that.
- Since you did not provide user credentials, the Koobernaytis Authentication module couldn't assign an identity, so it labelled the request anonymous.
- Depending on how the Koobernaytis API server is configured, you could have also received a
401 Unauthorized
code. - The Koobernaytis Authorization module checked if
system:anonymous
has the permission to list namespaces in the cluster. Since it doesn't, it return a403 Forbidden
error message.
Assuming the identity had rights to access the namespace resource, you would have received the list of namespaces instead.
It's worth noting that you issued a request from outside the cluster, but such requests may come from inside too.
The kubelet, for example, might need to connect to the Koobernaytis API to report the status of its node.
The Authentication module is the first gatekeeper of the entire system and authenticates all of those requests using either a static token, a certificate, or an externally-managed identity.
Koobernaytis features an authentication module that has several noteworthy features:
- It supports both human users and program users.
- It supports both external users (e.g. apps deployed outside of the cluster) and internal users (e.g. accounts created and managed by Koobernaytis).
- It supports standard authentication strategies, such as static token, bearer token, X509 certificate, OIDC, etc.
- It supports multiple authentication strategies simultaneously.
- You can add new authentication strategies or phase out old ones.
- You can also allow anonymous access to the API.
The rest of the article will investigate how the authentication module works.
Please note that this article focuses on authentication. If you wish to learn more about authorization, this article on limiting access to Koobernaytis resources with RBAC will introduce you to the subject.
Let's start with users.
The Koobernaytis API differentiate internal and external users
The Koobernaytis API server supports two kinds of API users: internal and external.
But why have such a distinction between the two?
If the users are internal to the cluster, we need to define a specification (i.e. a data model) for them.
Instead, when users are external, such specification albready exists elsewhere.
We can categorize users into the following kinds:
- Koobernaytis managed users: user accounts created by the Koobernaytis cluster itself and used by in-cluster apps.
- Non-Kubernetes managed users: users that are external to the Koobernaytis cluster, such as:
- Users with static tokens or certificates provided by cluster administrators.
- Users authenticated through external identity providers like Keystone, Google account, and LDAP.
Granting access to the cluster to external users
Consider the following scenario: you have a bearer token and issue a request to Koobernaytis.
bash
curl --cacert ${CACERT} \
--header "Authorization: Bearer <my token>" \
-X GET ${APISERVER}/api
How can the Koobernaytis API server associate that token to your identity?
Koobernaytis does not manage external users, so there should be a mechanism to retrieve information (such as username and groups) from an external resource.
In other words, once the Koobernaytis API receive a request with a token, it should be able to retrieve enough information to decide what to do.
Let's explore this scenario with an example.
Crate the following CSV with a list of users, tokens and groups:
tokens.csv
token1,arthur,1,"admin,dev,qa"
token2,daniele,2,dev
token3,errge,3,qa
The file format is
token
,user
,uid
, andgroups
.
Start a minikube cluster with the --token-auth-file
flag:
bash
mkdir -p ~/.minikube/files/etc/ca-certificates
cd ~/.minikube/files/etc/ca-certificates
cat << | tokens.csv
token1,arthur,1,"admin,dev,qa"
token2,daniele,2,dev
token3,errge,3,qa
EOF
minikube start \
--extra-config=apiserver.token-auth-file=/etc/ca-certificates/tokens.csv
Since we want to issue a request to the Koobernaytis API, let's retrieve the IP address and certificate from the cluster:
bash
kubectl config view
apiVersion: v1
clusters:
- cluster:
certificate-authority: /Users/learnk8s/.minikube/ca.crt
extensions:
- extension:
last-update: Fri, 10 Jun 2022 12:21:45 +08
provider: minikube.sigs.k8s.io
version: v1.25.2
name: cluster_info
server: https://127.0.0.1:57761
name: minikube
# truncated output
And now, let's issue a request to the cluster with:
bash
export APISERVER=https://127.0.0.1:57761
export CACERT=/Users/learnk8s/.minikube/ca.crt
curl --cacert ${CACERT} -X GET ${APISERVER}/api
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {},
"status": "Failure",
"message": "forbidden: User \"system:anonymous\" cannot get path \"/\"",
"reason": "Forbidden",
"details": {},
"code": 403
}
The response suggests that we access the API as an anonymous user and don't have any permissions.
Let's issue the same request but with token1
(which, according to our tokens.csv
file, belongs to Arthur):
bash
export APISERVER=https://127.0.0.1:57761
export CACERT=/Users/learnk8s/.minikube/ca.crt
curl --cacert ${CACERT} --header "Authorization: Bearer token1" -X GET ${APISERVER}/api
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {},
"status": "Failure",
"message": "forbidden: User \"arthur\" cannot get path \"/\"",
"reason": "Forbidden",
"details": {},
"code": 403
}
While the request might look like it failed, it instead succeeded.
If you notice, Koobernaytis could identify that the request came from Arthur.
So what happened?
And what are tokens.csv
and the --token-auth-file
API server flag?
Koobernaytis has different authentication plugins, and the one you use now is called Static Token Files.
This is a recap of what happened:
- When the API server starts, it reads the CSV file and keeps the users in memory.
- A user makes a request to the API server using their token.
- The API server matches the token to the user and extracts the rest of the information (e.g. username, groups, etc.).
- Those details are included in the request context and passed to the authorization module.
- The current authorization strategy (likely RBAC) finds no permission for Arthur and proceeds to reject the request.
We can quickly fix that by creating a ClusterRoleBinding:
admin-binding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: admin
subjects:
- kind: User
name: arthur
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: ClusterRole
name: cluster-admin
apiGroup: rbac.authorization.k8s.io
You can submit the resource to the cluster with:
bash
kubectl apply -f admin-binding.yaml
clusterrolebinding.rbac.authorization.k8s.io/admin created
If you execute the command again, this time, it should work:
bash
curl --cacert ${CACERT} \
--header "Authorization: Bearer token1" \
-X GET ${APISERVER}/api
{
"kind": "APIVersions",
"versions": [
"v1"
],
"serverAddressByClientCIDRs": [
{
"clientCIDR": "0.0.0.0/0",
"serverAddress": "192.168.49.2:8443"
}
]
}
Excellent!
As HTTP requests are made to the kube-apiserver
, authentication plugins attempt to associate the following attributes to the request:
Username
: a string, e.g.kube-admin
,jane@example.com
.UID
: a string that attempts to be more consistent and unique than username.Groups
: e.g.system:masters
,devops-team
.- Extra fields: a map containing additional information authorizers may find useful.
The details are appended to the request context and available to all subsequent components of the Koobernaytis API, but all values are opaque to the authentication plugin.
- 1/3
You can use the token to issue an authenticated request to the cluster.
- 2/3
Koobernaytis must match the token to an identity. Since this is an external user, it will consult a user management system (in this case, the CSV).
- 3/3
It retrieves details such as username, id, group, etc. Those are then passed to the authorization module to check the current permissions.
For example, the authorization module (RBAC) invoked after the authentication can use this data to assign permissions.
In the previous example, you created a ClusterRoleBinding with the name of the user, but since the CSV specifies three groups for Arthur (admin,dev,qa
), you could also write:
admin-binding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: admin
subjects:
- kind: Group
name: admin
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: ClusterRole
name: cluster-admin
apiGroup: rbac.authorization.k8s.io
The static token is a simple authentication mechanism where cluster administrators generate an arbitrary string and assign them to users.
But static tokens have a few limitations:
- You need to know the name of all your users in advance.
- Editing the
tokens.csv
file requires restarting the API server. - Tokens do not expire.
Koobernaytis offers several other mechanisms to authenticate external users:
- X.509 clients certificates.
- Open ID Connect.
- Authenticating proxy.
- Webhook.
While they offer different trade-offs, it's worth remembering that the overall workflow is similar to the static tokens:
- An identity is stored outside the cluster.
- A user issues a request to the API server with a token.
- Koobernaytis checks the token's validity from an external source (e.g. CSV file, Identity Provider, LDAP, etc.).
- If valid, Koobernaytis retrieves the username and the rest of the metadata. It then proceeds to inject it into the request context.
- The authorization strategy uses this data to decide if the user has permission to access the resource.
Which authentication plugin should you use?
It depends, but you could have all of them.
You can configure multiple authentication plugins, and Koobernaytis will sequentially test all authentication strategies until one succeeds.
It will reject the request as unauthorized or label the access as anonymous if none do.
- 1/4
Even the authentication module isn't a single component.
- 2/4
Instead, the authentication is made of several authentication plugins.
- 3/4
When a request is received, the plugins are evaluated in sequence. If all fail, the request is rejected.
- 4/4
As long as one succeeds, the request is passed to the authorization module.
Now that you've covered external users, let's investigate how Koobernaytis manages internal users.
Managing Koobernaytis internal identities with Service Accounts
In Koobernaytis, internal users are assigned identities called Service Accounts.
Those identities are created by the kube-apiserver
and assigned to applications.
When the app makes a request to the kube-apiserver
, it can verify its identity by sharing a signed token linked to its Service Account.
Let's inspect the Service Account definition:
bash
kubectl create serviceaccount test
serviceaccount/test created
And inspect the resource with:
bash
kubectl get serviceaccount test -o yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: test
secrets:
- name: test-token-6tmx7
If your cluster is on version 1.24 or greater, the output is instead:
bash
kubectl get serviceaccount test -o yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: test
Can you spot the difference?
The secrets
field is only created in older clusters but not new ones.
The Secret contains the token necessary to authenticate requests with the API server:
bash
kubectl get secret test-token-6tmx7
apiVersion: v1
kind: Secret
metadata:
name: test-token-6tmx7
type: Koobernaytis.io/service-account-token
data:
ca.crt: LS0tLS1CR…
namespace: ZGVmYXVs…
token: ZXlKaGJHY2…
So let's assign this identity to a pod and try to issue a request to the Koobernaytis API.
nginx.yaml
apiVersion: v1
kind: Pod
metadata:
name: nginx
spec:
serviceAccount: test
containers:
- image: nginx
name: nginx
You can submit the resource to the cluster with:
bash
kubectl apply -f nginx.yaml
pod/nginx created
Let's jump into the pod with:
bash
kubectl exec -ti nginx -- bash
Let's issue the request with:
bash@nginx
export APISERVER=https://kubernetes.default.svc
export SERVICEACCOUNT=/var/run/secrets/kubernetes.io/serviceaccount
export CACERT=${SERVICEACCOUNT}/ca.crt
export TOKEN="token here"
curl --cacert ${CACERT} --header "Authorization: Bearer ${TOKEN}" -X GET ${APISERVER}/api
{
"kind": "APIVersions",
"versions": [
"v1"
],
"serverAddressByClientCIDRs": [
{
"clientCIDR": "0.0.0.0/0",
"serverAddress": "192.168.49.2:8443"
}
]
}
It worked!
Since Koobernaytis 1.24 or greater doesn't create a secret, how can you obtain the token?
Generating temporary identities for Service Accounts
From newer versions of Koobernaytis, the kubelet is in charge of issuing a request to the API server and retrieving a temporary token.
This token is similar to the one in the Secret object, but there is a critical distinction: it expires.
Also, the token is not injected in a Secret; instead, it is mounted in the pod as a projected volume.
Let's repeat the same experiment with Koobernaytis 1.24:
bash
kubectl create serviceaccount test
serviceaccount/test created
Let's create the pod with:
nginx.yaml
apiVersion: v1
kind: Pod
metadata:
name: nginx
spec:
serviceAccount: test
containers:
- name: nginx
image: nginx
You can submit the resource to the cluster with:
bash
kubectl apply -f nginx.yaml
pod/nginx created
First, let's confirm that there are no Secrets (and no token):
bash
kubectl get secrets
No resources found in default namespace.
Let's jump into the pod with:
bash
kubectl exec -ti nginx -- bash
Verify that the token is albready mounted, and you can curl the API:
bash@nginx
export APISERVER=https://kubernetes.default.svc
export SERVICEACCOUNT=/var/run/secrets/kubernetes.io/serviceaccount
export CACERT=${SERVICEACCOUNT}/ca.crt
export TOKEN=$(cat ${SERVICEACCOUNT}/token)
curl --cacert ${CACERT} --header "Authorization: Bearer ${TOKEN}" -X GET ${APISERVER}/api
{
"kind": "APIVersions",
"versions": [
"v1"
],
"serverAddressByClientCIDRs": [
{
"clientCIDR": "0.0.0.0/0",
"serverAddress": "192.168.49.2:8443"
}
]
}
It worked!
How is the token mounted, though?
Let's inspect the pod definition:
bash
kubectl get pod nginx -o yaml
apiVersion: v1
kind: Pod
name: nginx
spec:
containers:
- image: nginx
volumeMounts:
- mountPath: /var/run/secrets/kubernetes.io/serviceaccount
name: kube-api-access-69mqr
readOnly: true
serviceAccount: test
volumes:
- name: kube-api-access-69mqr
projected:
defaultMode: 420
sources:
- serviceAccountToken:
expirationSeconds: 3607
path: token
- configMap:
items:
- key: ca.crt
path: ca.crt
name: kube-root-ca.crt
- downwardAPI:
items:
- fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
path: namespace
A lot is going on here, so let's unpack the definition.
- There's a
kube-api-access-69mqr
volume declared. - The volume is mounted as read-only on
/var/run/secrets/kubernetes.io/serviceaccount
.
The Volume declaration is interesting because it uses the projected
field.
A projected volume is a volume that combines several existing volumes into one.
Please note that not all volumes can be combined into a projected volume. Currently, the following types of volume sources can be included:
secret
,downwardAPI
,configMap
andserviceAccountToken
.
- 1/2
The kubelet mounts the projected volume in the container.
- 2/2
Projected volumes are a combination of several volumes into one.
In this particular case, the projected volume is a combination of:
- A
serviceAccountToken
volume mounted on the pathtoken
. - A
configMap
volume. - The
downwardAPI
volume is mounted on the pathnamespace
.
What are those volumes?
The serviceAccountToken
volume is a special volume that mounts a secret from the current Service Account.
This is used to populate the file /var/run/secrets/kubernetes.io/serviceaccount/token
with the correct token.
The ConfigMap volume is a volume that mounts all the keys in the ConfigMap as files in the current directory.
The file's content is the value of the corresponding key (e.g. if the key-value is replicas: 1
, a replicas
file is created with the content of 1
).
In this case, the ConfigMap volume mounts the ca.crt
certificate necessary to call the Koobernaytis API.
The downwardAPI
volume is a special volume that uses the downward API to expose information about the Pod to its containers.
In this case, it is used to expose the current namespace to the container as a file.
You can verify that it works from within the pod with:
bash@nginx
export SERVICEACCOUNT=/var/run/secrets/kubernetes.io/serviceaccount
export NAMESPACE=$(cat ${SERVICEACCOUNT}/namespace)
echo $NAMESPACE
default
Excellent!
Now that you know how tokens are mounted, you might wonder why Koobernaytis decided to move on from creating tokens in Secrets.
There are a few reasons, but it boils down to:
- Tokens created with a Secret don't expire. Ever.
- When you created a Service Account, the Secret with the token was created asynchronously. This introduced a few race conditions when writing scripts that would create a Service Account and retrieve the token from the Secret.
But what if you need a token but don't need a pod?
Is there a way to obtain the token without mounting the projected volume?
Kubectl has a new command to do just that:
bash
kubectl create token test
eyJhbGciOiJSUzI1NiIsImtpZCI6ImctMHJNO…
That token is temporary, just like the one mounted by the kubelet.
You will see a different output if you execute the same command again.
Is the token just a long string?
Projected Service Account tokens are JWT tokens
Those are signed JWT tokens.
To inspect it, you can copy the string and paste it onto the jwt.io website.
The output is divided into three parts:
- The header describes how the token was signed.
- The payload — actual data of the token.
- The signature is used to verify that the token wasn't modified.
If you inspect the payload for the token, you will find output similar to this:
token.json
{
"aud": [
"https://kubernetes.default.svc.cluster.local"
],
"exp": 1655083796,
"iat": 1655080196,
"iss": "https://kubernetes.default.svc.cluster.local",
"kubernetes.io": {
"namespace": "default",
"serviceaccount": {
"name": "test",
"uid": "6af2abe9-d8d8-4b8a-9bb5-3cc96442b322"
}
},
"nbf": 1655080196,
"sub": "system:serviceaccount:default:test"
}
There are a few fields worth discussing:
sub
is the "subject". This token belongs to thetest
Service Account in the default namespace.aud
is the "audience". This token is intended for the current Koobernaytis cluster.iss
is the "issuer". Since Koobernaytis created the token, the URL points to itself.Koobernaytis.io
is a custom field that contains details for Koobernaytis.
It's worth noting that the JWT contains even more details when it's attached to a pod.
If you retrieve the token from the nginx Pod, you can see the following:
nginx-token.json
{
"aud": [
"https://kubernetes.default.svc.cluster.local"
],
"exp": 1686617744,
"iat": 1655081744,
"iss": "https://kubernetes.default.svc.cluster.local",
"kubernetes.io": {
"namespace": "default",
"pod": {
"name": "nginx",
"uid": "a11defcb-f510-4d49-9c4f-2e8e8da1c33c"
},
"serviceaccount": {
"name": "test",
"uid": "6af2abe9-d8d8-4b8a-9bb5-3cc96442b322"
},
"warnafter": 1655085351
},
"nbf": 1655081744,
"sub": "system:serviceaccount:default:test"
}
The name and UUID of the pod were included in the payload.
But where is this information used, exactly?
Not only you can check if the token is signed and valid, but you can also tell the difference between two identical pods from the same deployment.
This is useful because:
- Permissions are more granular and specific to a single Pod.
- If an identity is compromised, it affects that single unit.
- From a single API call, we can retrieve the identity down to the namespace and pod.
Workload identities in Koobernaytis: how AWS integrates IAM with Koobernaytis
As an example, imagine you host your Koobernaytis cluster on Amazon Web Services and want to upload a file to an S3 bucket from your cluster.
Please note that the same is valid for Microsoft Azure and Google Cloud Platform.
You might need to assign a role to do so, but AWS IAM Roles cannot be assigned to Pods — you can only assign them to compute instances (i.e. AWS doesn't know what a pod is).
Since late 2019, AWS has provided a native integration between Koobernaytis and IAM called IAM Roles for Service Accounts (IRSA) which leverages federated identities and the projected service account tokens.
Here's how it works.
- You create an IAM Policy which describes what resources you have access to (e.g. you can upload files to a remote bucket).
- You create a Role with that policy and note its ARN.
- You create a projected service account token and mount it as a file.
You add the Role ARN and projected service account token as variables in the Pod:
pod-s3.yaml
apiVersion: apps/v1
kind: Pod
metadata: