Kubernetes Probe Configuration for Different Scenarios

Overview

Kubernetes is a powerful container orchestration platform, relies on various mechanisms to manage and maintain the health of containers in a cluster. Among the most vital features for ensuring the stability of applications running in a Kubernetes environment are liveness probes, readiness probes, and startup probes. These probes continuously monitor the state of containers and help the Kubernetes scheduler decide when to restart, route traffic to, or start containers.

In this blog, we will dive into configuring Kubernetes probes for different scenarios. Understanding how to use and configure probes appropriately will improve the availability, resilience, and overall reliability of applications in Kubernetes.

Pioneers in Cloud Consulting & Migration Services

Reduced infrastructural costs
Accelerated application deployment

Get Started

Kubernetes Probes

Probes are diagnostic checks that Kubernetes uses to assess the health of containers. There are three primary types of probes:

Liveness Probe: Ensures that the container is running. If the liveness probe fails, Kubernetes will restart the container.
Readiness Probe: Determines if a container is ready to accept traffic. A failed readiness probe prevents the container from receiving traffic.
Startup Probe: Checks if the container has started successfully. This is especially useful for containers that need more time to start, preventing them from being prematurely marked as unhealthy. Liveness and readiness probes are not initiated until the startup probe succeeds.

kub

Each probe can be configured using three methods:

HTTP GET: Kubernetes makes an HTTP request to a specified path on a container.
TCP Socket: Kubernetes tries to open a TCP socket on a specified port.
Exec: Kubernetes executes a command inside the container.

Basic Configuration for Probes

Before diving into specific use cases, let’s review how to define probes in a Kubernetes manifest.

Here’s an example of a simple Liveness Probe for a web application:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - name: my-app-container
        image: my-app:latest
        ports:
        - containerPort: 80
        livenessProbe:
          httpGet:
            path: /healthz
            port: 80
          initialDelaySeconds: 10
          periodSeconds: 30

apiVersion: apps/v1

kind: Deployment

metadata:

name: my-app

spec:

replicas: 3

selector:

matchLabels:

app: my-app

template:

metadata:

labels:

app: my-app

spec:

containers:

- name: my-app-container

image: my-app:latest

ports:

- containerPort: 80

livenessProbe:

httpGet:

path: /healthz

port: 80

initialDelaySeconds: 10

periodSeconds: 30

In this example, the livenessProbe checks the /healthz endpoint of the container every 30 seconds, starting 10 seconds after the container starts.

Scenario 1: Application That Takes Time to Start

Some applications, particularly those with complex startup processes like initializing databases or loading data into memory, can take a long time to prepare. In these cases, Kubernetes’ default probe settings may not suffice, and you may encounter premature failures.

Problem: Your application might be marked as unhealthy or unavailable if the startup time is longer than the default probe settings.

Solution: You can adjust the startupProbe to give your container sufficient time to start. A startup probe is useful because it prevents other probes (like readiness and liveness probes) from marking your container as unhealthy while it is still starting up.

Example of using a startupProbe:

startupProbe:
  httpGet:
    path: /healthz
    port: 80
  failureThreshold: 30
  periodSeconds: 5
  initialDelaySeconds: 15

startupProbe:

httpGet:

path: /healthz

port: 80

failureThreshold: 30

periodSeconds: 5

initialDelaySeconds: 15

In this case, the startupProbe checks every 5 seconds, with a failure threshold of 30 attempts. If the probe fails 30 times (or for 150 seconds), Kubernetes will restart the container. This ensures that Kubernetes doesn’t prematurely restart your container if it is still in the process of starting up.

Scenario 2: Short-Lived Jobs

Short-lived jobs require careful probe configuration, such as batch processes or jobs that are completed within minutes or seconds. If a readiness probe or liveness probe is not configured correctly, Kubernetes might incorrectly determine that the container is unhealthy and trigger unnecessary restarts.

Problem: If you have short-lived jobs that complete quickly, the readiness and liveness probes might cause Kubernetes to mark the container as unhealthy before it completes its job.

Solution: Configure the readinessProbe to be minimally invasive, or skip it if it’s unnecessary for jobs that don’t need traffic routing.

Example configuration for a short-lived job:

readinessProbe:
  httpGet:
    path: /readiness
    port: 8080
  initialDelaySeconds: 0
  periodSeconds: 5
  timeoutSeconds: 2

readinessProbe:

httpGet:

path: /readiness

port: 8080

initialDelaySeconds: 0

periodSeconds: 5

timeoutSeconds: 2

In this case, the readinessProbe is configured to start immediately and execute every 5 seconds, with a timeout of 2 seconds. Adjusting the initial delay and probe frequency ensures Kubernetes doesn’t prematurely assume the job has failed or is incomplete.

Alternatively, you could omit the readiness probe entirely for jobs that don’t require continuous health checks.

Scenario 3: Applications with External Dependencies

Many modern applications depend on external services like databases, caching systems, or third-party APIs. If any of these services go down, the application might become unresponsive. In such cases, the readiness and liveness probes should be configured to account for service dependencies.

Problem: If your application relies on external services that might experience intermittent failures or slowdowns, Kubernetes might incorrectly restart or fail to route traffic to your container.

Solution: Use readiness probes to ensure traffic isn’t routed to the container until all dependencies are met, and configure liveness probes to handle more resilient responses if a dependency fails.

Example configuration for an app depending on a database:

readinessProbe:
  exec:
    command:
      - "sh"
      - "-c"
      - "nc -zv db-host 5432"
  initialDelaySeconds: 5
  periodSeconds: 10

readinessProbe:

exec:

command:

- "sh"

- "-c"

- "nc -zv db-host 5432"

initialDelaySeconds: 5

periodSeconds: 10

Here, the readinessProbe uses an exec command to run an nc (netcat) check to see if the PostgreSQL database is reachable on port 5432. This ensures the app only accepts traffic when the database is available.

Scenario 4: Stateless Web Applications

Stateless web applications generally require simpler probes, which don’t depend on external services or complex initialization. For such cases, a liveness and readiness probe configuration can be straightforward.

Problem: While these apps don’t have complex requirements, improperly configured probes can still lead to unnecessary downtime or restarts.

Solution: Ensure that livenessProbe and readinessProbe are simple and configured with appropriate delays.

Example configuration for a stateless web app:

livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
  initialDelaySeconds: 10
  periodSeconds: 20

readinessProbe:
  httpGet:
    path: /readiness
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 10

livenessProbe:

httpGet:

path: /healthz

port: 8080

initialDelaySeconds: 10

periodSeconds: 20

readinessProbe:

httpGet:

path: /readiness

port: 8080

initialDelaySeconds: 5

periodSeconds: 10

In this case, the livenessProbe checks every 20 seconds, starting 10 seconds after the container starts. The readinessProbe ensures the container can handle traffic 5 seconds after startup.

Best Practices for Probe Configuration

Regardless of your specific scenario, there are some best practices to keep in mind when configuring Kubernetes probes:

Use Initial Delays: Always use initialDelaySeconds to allow your container time to start up properly before probes begin.
Tune Timeout and Period: Adjust the timeoutSeconds and periodSeconds based on how long your application typically takes to respond.
Use Multiple Probe Types: In more complex applications, you might use combinations of liveness, readiness, and startup probes to ensure robust health checks.
Monitor Logs and Metrics: Continuously monitor your probes’ success and failure rates to understand how your application behaves under different conditions.

Conclusion

Proper probe configuration in Kubernetes ensures your containers remain healthy, available, and scalable. By adjusting the probes to fit specific scenarios, such as applications with long startup times, short-lived jobs, external dependencies, or stateless apps, you can prevent unnecessary restarts and downtime while improving the overall health and responsiveness of your application.

Always fine-tune your probe configurations based on real-world behavior to ensure that Kubernetes can make the best decisions to maintain your cluster’s reliability.

Drop a query if you have any questions regarding Kubernetes Probes and we will get back to you quickly.

Making IT Networks Enterprise-ready – Cloud Management Services

Accelerated cloud migration
End-to-end view of the cloud environment

Get Started

About CloudThat

CloudThat is a leading provider of Cloud Training and Consulting services with a global presence in India, the USA, Asia, Europe, and Africa. Specializing in AWS, Microsoft Azure, GCP, VMware, Databricks, and more, the company serves mid-market and enterprise clients, offering comprehensive expertise in Cloud Migration, Data Platforms, DevOps, IoT, AI/ML, and more.

FAQs

1. What is the purpose of the startupProbe in Kubernetes?

ANS: – The startupProbe is used to determine if a container has started successfully. It’s especially useful for applications that require a longer time to initialize. By configuring this probe, Kubernetes will give your container sufficient time to start before checking its health with other probes, avoiding premature restarts.

2. How do I configure probes for a simple stateless web application?

ANS: – For stateless web applications, you can configure basic HTTP GET probes for both liveness and readiness. Use simple paths like /healthz for liveness and /readiness for readiness checks, and set appropriate delays (e.g., initialDelaySeconds) to ensure the container is ready to serve traffic.

WRITTEN BY Hridya Hari

Hridya Hari works as a Research Associate - Data and AIoT at CloudThat. She is a data science aspirant who is also passionate about cloud technologies. Her expertise also includes Exploratory Data Analysis.