Introduction
Understanding Kubernetes' concepts is key to running highly available applications.
This article will take you through the scenario of deploying a new version of a pod, and show how understanding the pod lifecycle and implementing a readiness probe will help you deploying new releases without downtime.
Without a readiness probe Kubernetes will try to guess when your pod is ready, and then schedule traffic to it. If the pod has latency between the point-in-time when the container is running and when it can handle traffic, this will cause transactions to be dropped, a.k.a. downtime. A typical situation where this may happen, is if your container has a heavy initialization sequence, like starting multiple processes, making consistency checks or downloading external content that it should serve.
Demonstration
All steps shown here can be found in this github repo. To generate latency we will use a pre-built docker image trondhindenes/k8sprobetester
. The author has also written a good article about probes.
In each step we will make a release update between v1
and v2
by preparing the environment (by deploying deployment-v1.yaml
) and then applying deployment-v2.yaml
while running a test (using curl
with a time-out of 1 second). The test result will show how each method of deploying a new version affected the availability of the application.
I you want to follow along in your environment the repo has all the needed instructions.
Step 1: naive approach
The v1
deployment in step1
has no readiness probe so Kubernetes will switch the traffic to the new pod when it has started. As we have introduced a latency of 10 seconds curl will time-out a few times, and recover once the new pod is ready to handle traffic.
This video demonstrates the transaction behavior (left window) and the way kubernetes replaces the v1
pod with v2
(top right window). Notice that the state of the new pod becomes “1/1” right away as Kubernetes “thinks” it is ready.
Step 2: with readiness probe
The v1
deployment in step2
has a readiness probe so Kubernetes will switch the traffic to the new pod when it’s probe says it is ready. As we have introduced a latency of 10 seconds the traffic will be routed to the previous version of the pod until the new pod is ready, and curl will not time-out.
spec:
containers:
- image: trondhindenes/k8sprobetester:latest
...
readinessProbe:
httpGet:
path: /healthz
port: 80
httpHeaders:
- name: Host
value: KubernetesLivenessProbe
initialDelaySeconds: 5
failureThreshold: 2
This video demonstrates the transaction behavior (left window) and the way kubernetes replaces the v1
pod with v2
(top right window). Notice that the state of the new pod first becomes “0/1”, and “1/1” with a delay (of the introduced latency. As long as the new version of the pod is not Kubernetes routes the traffic to the previous version, leading to a zero-downtime deployment.