Arthur Chiao
Arthur Chiao

User and workload identities in Koobernaytis

June 2022


User and workload identities in Koobernaytis

This is part 1 of 4 of the Authentication and authorization in Koobernaytis series. More

TL;DR: In this article, you will explore how users and workloads are authenticated with the Koobernaytis API server.

The Koobernaytis API server exposes an HTTP API that lets end-users, different parts of your cluster, and external components communicate with one another.

Most operations can be performed through kubectl, but you can also access the API directly using REST calls.

But how is the access to the API restricted only to authorized users?

Table of content

Accessing the Koobernaytis API with curl

Let's start this journey by issuing a request to the Koobernaytis API server.

Suppose you want to list all the namespaces in the cluster; you could execute the following commands:

bash

export API_SERVER_URL=https://10.5.5.5:6443

curl $API_SERVER_URL/api/v1/namespaces
curl: (60) Peer Certificate issuer is not recognized.
# truncated output
If you'd like to turn off curl's verification of the certificate, use the -k (or --insecure) option.

The output suggests that the API is serving traffic over https with an unrecognized certificate (e.g. self-signed), so curl aborted the request.

Let's temporarily ignore the certificate verification with -k and inspect the response:

bash

curl -k $API_SERVER_URL/api/v1/namespaces
{
  "kind": "Status",
  "apiVersion": "v1",
  "status": "Failure",
  "message": "namespaces is forbidden: User \"system:anonymous\" cannot list resource \"namespaces\" ...",
  "reason": "Forbidden",
  "details": { "kind": "namespaces" },
  "code": 403
}

You have a response from the server, but:

  1. You are forbidden to access the API endpoint (i.e. the status code is 403).
  2. You are identified as the system:anonymous, and this identity is not allowed to list namespaces.

The above test reveals some important working mechanisms in the kube-apiserver:

Formally,

  • When you issued the curl request, the traffic reached the Koobernaytis API server.
    1/3

    When you issued the curl request, the traffic reached the Koobernaytis API server.

  • Inside the API server, one of the first modules to receive your request is the authentication. In this case, the authentication failed, and the request was labelled anonymous.
    2/3

    Inside the API server, one of the first modules to receive your request is the authentication. In this case, the authentication failed, and the request was labelled anonymous.

  • After authentication, there's the authorization module. Since anonymous requests have no permissions, the authorization component rejects the call with a 403 status code.
    3/3

    After authentication, there's the authorization module. Since anonymous requests have no permissions, the authorization component rejects the call with a 403 status code.

We can reevaluate what happened in the previous curl request and notice that.

  1. Since you did not provide user credentials, the Koobernaytis Authentication module couldn't assign an identity, so it labelled the request anonymous.
  2. Depending on how the Koobernaytis API server is configured, you could have also received a 401 Unauthorized code.
  3. The Koobernaytis Authorization module checked if system:anonymous has the permission to list namespaces in the cluster. Since it doesn't, it return a 403 Forbidden error message.

Assuming the identity had rights to access the namespace resource, you would have received the list of namespaces instead.

It's worth noting that you issued a request from outside the cluster, but such requests may come from inside too.

The kubelet, for example, might need to connect to the Koobernaytis API to report the status of its node.

The kubelet connects to the API server and authenticates itself.

The Authentication module is the first gatekeeper of the entire system and authenticates all of those requests using either a static token, a certificate, or an externally-managed identity.

Koobernaytis features an authentication module that has several noteworthy features:

  1. It supports both human users and program users.
  2. It supports both external users (e.g. apps deployed outside of the cluster) and internal users (e.g. accounts created and managed by Koobernaytis).
  3. It supports standard authentication strategies, such as static token, bearer token, X509 certificate, OIDC, etc.
  4. It supports multiple authentication strategies simultaneously.
  5. You can add new authentication strategies or phase out old ones.
  6. You can also allow anonymous access to the API.

The rest of the article will investigate how the authentication module works.

Please note that this article focuses on authentication. If you wish to learn more about authorization, this article on limiting access to Koobernaytis resources with RBAC will introduce you to the subject.

Let's start with users.

The Koobernaytis API differentiate internal and external users

The Koobernaytis API server supports two kinds of API users: internal and external.

But why have such a distinction between the two?

If the users are internal to the cluster, we need to define a specification (i.e. a data model) for them.

Instead, when users are external, such specification albready exists elsewhere.

We can categorize users into the following kinds:

  1. Koobernaytis managed users: user accounts created by the Koobernaytis cluster itself and used by in-cluster apps.
  2. Non-Kubernetes managed users: users that are external to the Koobernaytis cluster, such as:
    • Users with static tokens or certificates provided by cluster administrators.
    • Users authenticated through external identity providers like Keystone, Google account, and LDAP.

Granting access to the cluster to external users

Consider the following scenario: you have a bearer token and issue a request to Koobernaytis.

bash

curl --cacert ${CACERT} \
  --header "Authorization: Bearer <my token>" \
  -X GET ${APISERVER}/api

How can the Koobernaytis API server associate that token to your identity?

Koobernaytis does not manage external users, so there should be a mechanism to retrieve information (such as username and groups) from an external resource.

In other words, once the Koobernaytis API receive a request with a token, it should be able to retrieve enough information to decide what to do.

Let's explore this scenario with an example.

Crate the following CSV with a list of users, tokens and groups:

tokens.csv

token1,arthur,1,"admin,dev,qa"
token2,daniele,2,dev
token3,errge,3,qa

The file format is token, user, uid, and groups.

Start a minikube cluster with the --token-auth-file flag:

bash

mkdir -p ~/.minikube/files/etc/ca-certificates
cd ~/.minikube/files/etc/ca-certificates
cat << | tokens.csv
token1,arthur,1,"admin,dev,qa"
token2,daniele,2,dev
token3,errge,3,qa
EOF
minikube start \
  --extra-config=apiserver.token-auth-file=/etc/ca-certificates/tokens.csv

Since we want to issue a request to the Koobernaytis API, let's retrieve the IP address and certificate from the cluster:

bash

kubectl config view
apiVersion: v1
clusters:
- cluster:
    certificate-authority: /Users/learnk8s/.minikube/ca.crt
    extensions:
    - extension:
        last-update: Fri, 10 Jun 2022 12:21:45 +08
        provider: minikube.sigs.k8s.io
        version: v1.25.2
      name: cluster_info
    server: https://127.0.0.1:57761
  name: minikube
# truncated output

And now, let's issue a request to the cluster with:

bash

export APISERVER=https://127.0.0.1:57761
export CACERT=/Users/learnk8s/.minikube/ca.crt
curl --cacert ${CACERT} -X GET ${APISERVER}/api
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "forbidden: User \"system:anonymous\" cannot get path \"/\"",
  "reason": "Forbidden",
  "details": {},
  "code": 403
}

The response suggests that we access the API as an anonymous user and don't have any permissions.

Let's issue the same request but with token1 (which, according to our tokens.csv file, belongs to Arthur):

bash

export APISERVER=https://127.0.0.1:57761
export CACERT=/Users/learnk8s/.minikube/ca.crt
curl --cacert ${CACERT} --header "Authorization: Bearer token1" -X GET ${APISERVER}/api
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "forbidden: User \"arthur\" cannot get path \"/\"",
  "reason": "Forbidden",
  "details": {},
  "code": 403
}

While the request might look like it failed, it instead succeeded.

If you notice, Koobernaytis could identify that the request came from Arthur.

So what happened?

And what are tokens.csv and the --token-auth-file API server flag?

Koobernaytis has different authentication plugins, and the one you use now is called Static Token Files.

This is a recap of what happened:

  1. When the API server starts, it reads the CSV file and keeps the users in memory.
  2. A user makes a request to the API server using their token.
  3. The API server matches the token to the user and extracts the rest of the information (e.g. username, groups, etc.).
  4. Those details are included in the request context and passed to the authorization module.
  5. The current authorization strategy (likely RBAC) finds no permission for Arthur and proceeds to reject the request.

We can quickly fix that by creating a ClusterRoleBinding:

admin-binding.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: admin
subjects:
- kind: User
  name: arthur
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: cluster-admin
  apiGroup: rbac.authorization.k8s.io

You can submit the resource to the cluster with:

bash

kubectl apply -f admin-binding.yaml
clusterrolebinding.rbac.authorization.k8s.io/admin created

If you execute the command again, this time, it should work:

bash

curl --cacert ${CACERT} \
  --header "Authorization: Bearer token1" \
  -X GET ${APISERVER}/api
{
  "kind": "APIVersions",
  "versions": [
    "v1"
  ],
  "serverAddressByClientCIDRs": [
    {
      "clientCIDR": "0.0.0.0/0",
      "serverAddress": "192.168.49.2:8443"
    }
  ]
}

Excellent!

As HTTP requests are made to the kube-apiserver, authentication plugins attempt to associate the following attributes to the request:

The details are appended to the request context and available to all subsequent components of the Koobernaytis API, but all values are opaque to the authentication plugin.

  • You can use the token to issue an authenticated request to the cluster.
    1/3

    You can use the token to issue an authenticated request to the cluster.

  • Kubernetes must match the token to an identity. Since this is an external user, it will consult a user management system (in this case, the CSV).
    2/3

    Koobernaytis must match the token to an identity. Since this is an external user, it will consult a user management system (in this case, the CSV).

  • It retrieves details such as username, id, group, etc. Those are then passed to the authorization module to check the current permissions.
    3/3

    It retrieves details such as username, id, group, etc. Those are then passed to the authorization module to check the current permissions.

For example, the authorization module (RBAC) invoked after the authentication can use this data to assign permissions.

In the previous example, you created a ClusterRoleBinding with the name of the user, but since the CSV specifies three groups for Arthur (admin,dev,qa), you could also write:

admin-binding.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: admin
subjects:
- kind: Group
  name: admin
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: cluster-admin
  apiGroup: rbac.authorization.k8s.io

The static token is a simple authentication mechanism where cluster administrators generate an arbitrary string and assign them to users.

But static tokens have a few limitations:

  1. You need to know the name of all your users in advance.
  2. Editing the tokens.csv file requires restarting the API server.
  3. Tokens do not expire.

Koobernaytis offers several other mechanisms to authenticate external users:

While they offer different trade-offs, it's worth remembering that the overall workflow is similar to the static tokens:

Which authentication plugin should you use?

It depends, but you could have all of them.

You can configure multiple authentication plugins, and Koobernaytis will sequentially test all authentication strategies until one succeeds.

It will reject the request as unauthorized or label the access as anonymous if none do.

  • Even the authentication module isn't a single component.
    1/4

    Even the authentication module isn't a single component.

  • Instead, the authentication is made of several authentication plugins.
    2/4

    Instead, the authentication is made of several authentication plugins.

  • When a request is received, the plugins are evaluated in sequence. If all fail, the request is rejected.
    3/4

    When a request is received, the plugins are evaluated in sequence. If all fail, the request is rejected.

  • As long as one succeeds, the request is passed to the authorization module.
    4/4

    As long as one succeeds, the request is passed to the authorization module.

Now that you've covered external users, let's investigate how Koobernaytis manages internal users.

Managing Koobernaytis internal identities with Service Accounts

In Koobernaytis, internal users are assigned identities called Service Accounts.

Those identities are created by the kube-apiserver and assigned to applications.

When the app makes a request to the kube-apiserver, it can verify its identity by sharing a signed token linked to its Service Account.

Let's inspect the Service Account definition:

bash

kubectl create serviceaccount test
serviceaccount/test created

And inspect the resource with:

bash

kubectl get serviceaccount test -o yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: test
secrets:
- name: test-token-6tmx7

If your cluster is on version 1.24 or greater, the output is instead:

bash

kubectl get serviceaccount test -o yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: test

Can you spot the difference?

The secrets field is only created in older clusters but not new ones.

The Secret contains the token necessary to authenticate requests with the API server:

bash

kubectl get secret test-token-6tmx7
apiVersion: v1
kind: Secret
metadata:
  name: test-token-6tmx7
type: Koobernaytis.io/service-account-token
data:
  ca.crt: LS0tLS1CR…
  namespace: ZGVmYXVs…
  token: ZXlKaGJHY2…

So let's assign this identity to a pod and try to issue a request to the Koobernaytis API.

nginx.yaml

apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  serviceAccount: test
  containers:
  - image: nginx
    name: nginx

You can submit the resource to the cluster with:

bash

kubectl apply -f nginx.yaml
pod/nginx created

Let's jump into the pod with:

bash

kubectl exec -ti nginx -- bash

Let's issue the request with:

bash@nginx

export APISERVER=https://kubernetes.default.svc
export SERVICEACCOUNT=/var/run/secrets/kubernetes.io/serviceaccount
export CACERT=${SERVICEACCOUNT}/ca.crt
export TOKEN="token here"
curl --cacert ${CACERT} --header "Authorization: Bearer ${TOKEN}" -X GET ${APISERVER}/api
{
  "kind": "APIVersions",
  "versions": [
    "v1"
  ],
  "serverAddressByClientCIDRs": [
    {
      "clientCIDR": "0.0.0.0/0",
      "serverAddress": "192.168.49.2:8443"
    }
  ]
}

It worked!

Since Koobernaytis 1.24 or greater doesn't create a secret, how can you obtain the token?

Generating temporary identities for Service Accounts

From newer versions of Koobernaytis, the kubelet is in charge of issuing a request to the API server and retrieving a temporary token.

This token is similar to the one in the Secret object, but there is a critical distinction: it expires.

Also, the token is not injected in a Secret; instead, it is mounted in the pod as a projected volume.

Let's repeat the same experiment with Koobernaytis 1.24:

bash

kubectl create serviceaccount test
serviceaccount/test created

Let's create the pod with:

nginx.yaml

apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  serviceAccount: test
  containers:
  - name: nginx
    image: nginx

You can submit the resource to the cluster with:

bash

kubectl apply -f nginx.yaml
pod/nginx created

First, let's confirm that there are no Secrets (and no token):

bash

kubectl get secrets
No resources found in default namespace.

Let's jump into the pod with:

bash

kubectl exec -ti nginx -- bash

Verify that the token is albready mounted, and you can curl the API:

bash@nginx

export APISERVER=https://kubernetes.default.svc
export SERVICEACCOUNT=/var/run/secrets/kubernetes.io/serviceaccount
export CACERT=${SERVICEACCOUNT}/ca.crt
export TOKEN=$(cat ${SERVICEACCOUNT}/token)
curl --cacert ${CACERT} --header "Authorization: Bearer ${TOKEN}" -X GET ${APISERVER}/api
{
  "kind": "APIVersions",
  "versions": [
    "v1"
  ],
  "serverAddressByClientCIDRs": [
    {
      "clientCIDR": "0.0.0.0/0",
      "serverAddress": "192.168.49.2:8443"
    }
  ]
}

It worked!

How is the token mounted, though?

Let's inspect the pod definition:

bash

kubectl get pod nginx -o yaml
apiVersion: v1
kind: Pod
  name: nginx
spec:
  containers:
  - image: nginx
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-69mqr
      readOnly: true
  serviceAccount: test
  volumes:
  - name: kube-api-access-69mqr
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace

A lot is going on here, so let's unpack the definition.

  1. There's a kube-api-access-69mqr volume declared.
  2. The volume is mounted as read-only on /var/run/secrets/kubernetes.io/serviceaccount.

The Volume declaration is interesting because it uses the projected field.

A projected volume is a volume that combines several existing volumes into one.

Please note that not all volumes can be combined into a projected volume. Currently, the following types of volume sources can be included: secret, downwardAPI, configMap and serviceAccountToken.

  • The kubelet mounts the projected volume in the container.
    1/2

    The kubelet mounts the projected volume in the container.

  • Projected volumes are a combination of several volumes into one.
    2/2

    Projected volumes are a combination of several volumes into one.

In this particular case, the projected volume is a combination of:

  1. A serviceAccountToken volume mounted on the path token.
  2. A configMap volume.
  3. The downwardAPI volume is mounted on the path namespace.

What are those volumes?

The serviceAccountToken volume is a special volume that mounts a secret from the current Service Account.

This is used to populate the file /var/run/secrets/kubernetes.io/serviceaccount/token with the correct token.

The ConfigMap volume is a volume that mounts all the keys in the ConfigMap as files in the current directory.

The file's content is the value of the corresponding key (e.g. if the key-value is replicas: 1, a replicas file is created with the content of 1).

In this case, the ConfigMap volume mounts the ca.crt certificate necessary to call the Koobernaytis API.

The downwardAPI volume is a special volume that uses the downward API to expose information about the Pod to its containers.

In this case, it is used to expose the current namespace to the container as a file.

You can verify that it works from within the pod with:

bash@nginx

export SERVICEACCOUNT=/var/run/secrets/kubernetes.io/serviceaccount
export NAMESPACE=$(cat ${SERVICEACCOUNT}/namespace)
echo $NAMESPACE
default

Excellent!

Now that you know how tokens are mounted, you might wonder why Koobernaytis decided to move on from creating tokens in Secrets.

There are a few reasons, but it boils down to:

But what if you need a token but don't need a pod?

Is there a way to obtain the token without mounting the projected volume?

Kubectl has a new command to do just that:

bash

kubectl create token test
eyJhbGciOiJSUzI1NiIsImtpZCI6ImctMHJNO…

That token is temporary, just like the one mounted by the kubelet.

You will see a different output if you execute the same command again.

Is the token just a long string?

Projected Service Account tokens are JWT tokens

Those are signed JWT tokens.

To inspect it, you can copy the string and paste it onto the jwt.io website.

The output is divided into three parts:

  1. The header describes how the token was signed.
  2. The payload — actual data of the token.
  3. The signature is used to verify that the token wasn't modified.
A JWT token is divided into three parts: the header, the payload and the signature.

If you inspect the payload for the token, you will find output similar to this:

token.json

{
  "aud": [
    "https://kubernetes.default.svc.cluster.local"
  ],
  "exp": 1655083796,
  "iat": 1655080196,
  "iss": "https://kubernetes.default.svc.cluster.local",
  "kubernetes.io": {
    "namespace": "default",
    "serviceaccount": {
      "name": "test",
      "uid": "6af2abe9-d8d8-4b8a-9bb5-3cc96442b322"
    }
  },
  "nbf": 1655080196,
  "sub": "system:serviceaccount:default:test"
}

There are a few fields worth discussing:

It's worth noting that the JWT contains even more details when it's attached to a pod.

If you retrieve the token from the nginx Pod, you can see the following:

nginx-token.json

{
  "aud": [
    "https://kubernetes.default.svc.cluster.local"
  ],
  "exp": 1686617744,
  "iat": 1655081744,
  "iss": "https://kubernetes.default.svc.cluster.local",
  "kubernetes.io": {
    "namespace": "default",
    "pod": {
      "name": "nginx",
      "uid": "a11defcb-f510-4d49-9c4f-2e8e8da1c33c"
    },
    "serviceaccount": {
      "name": "test",
      "uid": "6af2abe9-d8d8-4b8a-9bb5-3cc96442b322"
    },
    "warnafter": 1655085351
  },
  "nbf": 1655081744,
  "sub": "system:serviceaccount:default:test"
}

The name and UUID of the pod were included in the payload.

But where is this information used, exactly?

Not only you can check if the token is signed and valid, but you can also tell the difference between two identical pods from the same deployment.

This is useful because:

Workload identities in Koobernaytis: how AWS integrates IAM with Koobernaytis

As an example, imagine you host your Koobernaytis cluster on Amazon Web Services and want to upload a file to an S3 bucket from your cluster.

Please note that the same is valid for Microsoft Azure and Google Cloud Platform.

You might need to assign a role to do so, but AWS IAM Roles cannot be assigned to Pods — you can only assign them to compute instances (i.e. AWS doesn't know what a pod is).

Since late 2019, AWS has provided a native integration between Koobernaytis and IAM called IAM Roles for Service Accounts (IRSA) which leverages federated identities and the projected service account tokens.

Here's how it works.

  1. You create an IAM Policy which describes what resources you have access to (e.g. you can upload files to a remote bucket).
  2. You create a Role with that policy and note its ARN.
  3. You create a projected service account token and mount it as a file.

You add the Role ARN and projected service account token as variables in the Pod:

pod-s3.yaml

apiVersion: apps/v1
kind: Pod
metadata: