How to create and debug a Ku­ber­netes State­fulSet

Each pod in a Ku­ber­netes State­fulSet has a unique and stable hostname as well as a complete DNS identity. This ID is retained across restarts and scaling.

What are Ku­ber­netes State­fulSets?

Ku­ber­netes State­fulSets are special entities to manage ap­pli­ca­tions that have special re­quire­ments for per­sis­tent data and fixed iden­ti­ties. These sets ensure that pods with replicas are started in a specific order. They assign each Ku­ber­netes pod a unique ID and access to per­sis­tent storage. Such functions are useful for databases that rely on stable re­la­tion­ships between instances and that need to store per­sis­tent data. Therefore, State­fulSets represent an effective solution for the or­ches­tra­tion and operation of complex ap­pli­ca­tions in Ku­ber­netes.

How to create a Ku­ber­netes State­fulSet

We start by defining a YAML con­fig­u­ra­tion file in which we specify the desired prop­er­ties of the State­fulSet and create Ku­ber­netes pods. After con­fig­u­ra­tion, the State­fulSet con­tin­u­ous­ly monitors the cluster status and whether the pre­de­fined number of pods is always running and available. In the event of a pod failure or removal from the node, the State­fulSet au­to­mat­i­cal­ly rec­og­nizes the situation. It initiates the de­ploy­ment of a new pod with the same unique ID. This new pod is connected to the existing per­sis­tent storage and receives the identical con­fig­u­ra­tion as the original pod. This includes resource requests and limits.

This precise handling of pods and their identity is critical so that clients pre­vi­ous­ly served by the un­avail­able pod can be redi­rect­ed to the new pod without in­ter­rup­tion. Access to per­sis­tent storage ensures that op­er­a­tions continue to function smoothly.

Here’s a complete YAML file that il­lus­trates the steps for creating a Ku­ber­netes State­fulSet for Nginx:

apiVersion: apps/v1
kind: StatefulSet
metadata:
    name: nginx-statefulset
spec:
    serviceName: nginx
    replicas: 3
    selector:
        matchLabels:
            app: nginx
    template:
        metadata:
            labels:
                app: nginx
        spec:
            containers:
            - name: nginx
                image: nginx:latest
                volumeMounts:
                - name: nginx-config
                    mountPath: /etc/nginx/conf.d
                - name: nginx-html
                    mountPath: /usr/share/nginx/html
    volumeClaimTemplates:
    - metadata:
            name: nginx-html
        spec:
            accessModes: [ "ReadWriteOnce" ]
            resources:
                requests:
                    storage: 1Gi
    volumeClaimTemplates:
    - metadata:
            name: nginx-config
        spec:
            accessModes: [ "ReadWriteOnce" ]
            resources:
                requests:
                    storage: 1Gi

This YAML document defines a Ku­ber­netes State­fulSet for Nginx with three replicas. It uses a service object called nginx and labels to correctly identify the pods. The latter use the latest Nginx image and have two volumes for con­fig­u­ra­tion and HTML files. The Vol­ume­ClaimTem­plates secure per­sis­tent storage for these volumes with a size of 1 gigabyte and allow Read­WriteOnce access.

Debugging State­fulSets

Debugging State­fulSets in Ku­ber­netes requires specific steps to verify that the pods are ini­tial­ized correctly and that you can identify and fix errors if necessary.

Step 1: List the pods

Before you start debugging State­fulSets, you should check the status of the pods.

Open the command line and use the following command to list all pods in the desired State­fulSet:

kubectl get pods -l app=statefulset-label

Output:

NAME                         READY      STATUS    RESTARTS   AGE
nginx-statefulset-0           1/1      Running       0       2m
nginx-statefulset-1           1/1      Running       0       1m
nginx-statefulset-2           1/1      Running       0       1m

• NAME: Each pod is given a unique name based on the naming scheme of the State­fulSet and a con­sec­u­tive number.
• READY: Indicates how many of the desired con­tain­ers are ready in the pod. In the example, each pod has one container and 1/1 means that the container is ready for use.
• STATUS: This indicates the current status of the pod.
• RESTARTS: Shows how often the container has been restarted in the pod. A value of 0 means that there have been no restarts so far.
• AGE: Indicates how long the pod has been running.

These are the status messages that indicate errors:

• Failed: One or more con­tain­ers in the pod caused an error that resulted in the failure of the pod. This can have various reasons, such as missing de­pen­den­cies or con­fig­u­ra­tion problems.
• Unknown: The state of the pod could not be de­ter­mined. It could indicate a problem with the com­mu­ni­ca­tion between the Ku­ber­netes cluster and the pod. These include network problems, missing au­tho­riza­tions or other factors.

In both cases, it’s important to use accurate di­ag­nos­tic tools such as checking pod logs or the kubectl describe pod command to get more details and determine the causes of the errors.

Step 2: Debug in­di­vid­ual pods

Adding an­no­ta­tions to a pod can be a useful debugging tool to influence the ini­tial­iza­tion process or trigger special actions.

First, you need to identify the name of the pod you want to debug.

kubectl get pods

Now enter the following command in the terminal to define the an­no­ta­tion for the selected pod:

kubectl annotate pods [pod-name] pod.alpha.kubernetes.io/initialized="false" --overwrite

Replace [pod-name] with the actual name of the pod. This an­no­ta­tion sets the ini­tial­iza­tion status to false, which means that the pod is marked as unini­tial­ized.

Monitor the pod to see how the an­no­ta­tion affects its behavior. In par­tic­u­lar, you can check the pod events and logs:

kubectl describe pod [pod-name]
kubectl logs [pod-name]

Look for events or log output that could cause problems during the ini­tial­iza­tion process. When debugging is complete and you want to restore the normal ini­tial­iza­tion process, set the an­no­ta­tion back to true.

Step 3: Step-by-step ini­tial­iza­tion

If debugging the pod using the tech­niques described above was not suc­cess­ful, this could indicate race con­di­tions during the bootstrap process of the State­fulSet. To overcome this issue, you can specify initialized="false" in the manifest of the Ku­ber­netes State­fulSet and then create it with this an­no­ta­tion in the cluster.

apiVersion: apps/v1
kind: StatefulSet
metadata:
    name: [statefulset-name]
spec:
    template:
        metadata:
            annotations:
                pod.alpha.kubernetes.io/initialized: "false"
        ...

Apply the updated manifest to your Ku­ber­netes cluster:

kubectl apply -f statefulset.yaml

Inspect the pods and identify any sources of error. Carry out the necessary debugging measures based on the observed events and logs. If necessary, delete pods with kubectl delete statefulsets or kubectl delete service.

After you have completed debugging, you can remove the ini­tial­ized an­no­ta­tion and update the Ku­ber­netes State­fulSet in the cluster:

kubectl annotate pods [pod-name] pod.alpha.kubernetes.io/initialized="true" --overwrite

The command sets the ini­tial­ized an­no­ta­tion of a pod to true and ensures that existing values are over­writ­ten. After you have checked the first pod, the Ku­ber­netes State­fulSet will au­to­mat­i­cal­ly ini­tial­ize the next pod. You can then repeat the debugging steps for each ad­di­tion­al pod in the State­fulSet.

In the Ku­ber­netes tutorial, you’ll find detailed and practical in­for­ma­tion on setting up a Ku­ber­netes cluster.