Graceful shutdown in Koobernaytis

April 2024

TL;DR: In this article, you will learn how to prevent broken connections when a Pod starts or shuts down. You will also learn how to shut down long-running tasks and connections gracefully.

In Koobernaytis, creating and deleting Pods is one of the most common tasks.

Pods are created when you execute a rolling update, scale deployments, for every new release, for every job and cron job, etc.

However, pods are also deleted and recreated after evictions—when you mark a node as not schedulable, for example.

If the nature of those pods is so ephemeral, what happens when a pod is in the middle of responding to a request but is told to shut down?

Is the request completed before shutdown?

What about subsequent requests? Are those redirected somewhere else?

Table of contents

• What happens when you create a Pod in Koobernaytis
• The kubelet creates and look after the pods
• Pods, Services and Endpoints
• Endpoints are the Koobernaytis currency
• What happens when you delete a Pod
• How to gracefully shut down pods
• Graceful shutdown with a preStop hook
• Long graceful shutdowns and cluster autoscaling
• Graceful shutdown for long-lived connections and long-running tasks

What happens when you create a Pod in Koobernaytis

Before discussing what happens when a Pod is deleted, it's necessary to discuss what happens when a Pod is created.

Let's assume you want to create the following Pod in your cluster:

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  containers:
    - name: web
      image: nginx
      ports:
        - name: web
          containerPort: 80

You can submit the YAML definition to the cluster with the following command:

kubectl apply -f pod.yaml

When you enter the command, kubectl submits the Pod definition to the Koobernaytis API.

This is where the journey begins.

The API receives and inspects the Pod definition and subsequently stored in the database — etcd.

The Pod is also added to the Scheduler's queue.

The Scheduler:

1. Inspects the definition.
2. Collects details about the workload, such as CPU and memory requests, and then
3. Decides which Node is best suited to run it (through a process called Filters and Predicates).

At the end of the process:

• The Pod is marked as Scheduled in etcd.
• The Pod has a Node assigned to it.
• The state of the Pod is stored in etcd.

But the Pod still does not exist.

The previous tasks happened in the control plane, and the state is stored in the database.

So who is creating the Pod in your Nodes?

The kubelet creates and look after the pods

The kubelet's job is to poll the control plane for updates.

You can imagine the kubelet relentlessly asking the control plane: "I look after the worker Node 1; is there any new Pod for me?".

When there is a Pod, the kubelet creates it.

Sort of.

The kubelet doesn't create the Pod by itself. Instead, it delegates the work to three other components:

1. The Container Runtime Interface (CRI) creates the containers for the Pod.
2. The Container Network Interface (CNI) connects the containers to the cluster network and assigns IP addresses.
3. The Container Storage Interface (CSI) mounts volumes in your containers.

In most cases, the Container Runtime Interface (CRI) is doing a similar job to:

docker run -d <my-container-image>

The Container Networking Interface (CNI) is a bit more interesting because it is in charge of:

1. Generating a valid IP address for the Pod.
2. Connecting the container to the rest of the network.

As you can imagine, several ways exist to connect the container to the network and assign a valid IP address (you could choose between IPv4 or IPv6 or multiple IP addresses).

If you