Progressive Delivery at Scale: Automated Canary Releases with Argo Rollouts and Kubernetes

Introduction

Cloud platforms have made it much easier to build highly available applications. Teams deploy across multiple Availability Zones, configure Auto Scaling, add health checks, create backups, and put monitoring around critical services.

But there is an important gap between having resilience mechanisms in place and knowing that those mechanisms will actually work when something breaks.

A Kubernetes deployment may be configured to restart failed pods. An Auto Scaling group may be able to replace an unhealthy EC2 instance. A load balancer may be configured to remove unhealthy targets. On paper, everything looks correct.

The real question is: What happens when we deliberately test those assumptions?

This is where Chaos Engineering becomes valuable.

Rather than waiting for an unexpected outage to reveal a weakness, teams can introduce controlled failures and observe how the system responds. AWS Fault Injection Service (AWS FIS) provides a way to run these experiments against supported AWS resources while applying safeguards around the blast radius.

When combined with Kubernetes, CloudWatch, Prometheus, automated recovery mechanisms, and SLO-based monitoring, Chaos Engineering turns resilience from something that is documented into something that can be tested and measured.

Architecture Overview

Architecture Explanation:

The architecture follows a controlled resilience-testing workflow:

Define Experiment → Apply Safeguards → Inject Failure → Observe System → Validate Recovery → Analyze Results → Improve Architecture

AWS Fault Injection Service acts as the experiment engine. The target environment can include Amazon EC2, Amazon EKS, Auto Scaling workloads, and other supported resources.

During an experiment, observability systems capture what is happening at both the infrastructure and application layers. Recovery mechanisms then respond to the injected failure, allowing the engineering team to verify whether the system behaves as expected.

The important part of the architecture is the feedback loop. An experiment should not end when the failure is injected. It ends when the team understands how the platform behaved, whether recovery worked, and what needs to change.

The Shift: From Failure Avoidance to Failure Readiness

Traditional infrastructure engineering naturally focuses on prevention.

Teams build redundancy, configure alarms, distribute workloads across Availability Zones, and create disaster recovery procedures. All of these controls are necessary.

But distributed systems eventually encounter failures that were not part of the original design assumptions.

A network path becomes unreliable. A dependency starts returning errors. A node disappears. A downstream service slows down. A queue grows unexpectedly.

Instead of trying to predict every possible failure, Chaos Engineering takes a different approach:

“Let’s test what happens when something fails.”

That simple change in mindset can expose gaps that are difficult to identify through architecture reviews or documentation alone.

For example:

• Does Kubernetes recover the workload within the expected time?
• Does Auto Scaling replace failed infrastructure quickly enough?
• Does the application tolerate increased latency from a dependency?
• Does traffic move away from unhealthy targets?
• Do retries amplify the problem or help the application recover?
• Does the operations team receive the right alert?
• Can the service recover within the defined RTO?

These are not theoretical questions once the experiment is running. The platform has to demonstrate the answer.

Core Pillars of Chaos Engineering
1. Define the Steady State First

Before introducing a failure, the team needs to know what healthy behavior looks like.

That could mean:

• API availability above the defined SLO
• P95 latency below an agreed threshold
• Error rate within the acceptable range
• Successful business transactions remaining above the target

The steady state becomes the baseline against which the experiment is evaluated.

A useful experiment should therefore start with measurable expectations rather than a vague goal such as “see what happens.”

 2. Inject Failures Deliberately

Chaos should never mean introducing random disruption without boundaries.

The failure scenario should be specific and the affected resources should be known in advance.

Examples include:

• Terminating a selected EC2 instance
• Restarting a non-critical Kubernetes pod
• Introducing CPU or memory pressure
• Creating network latency
• Testing the failure of a dependency
• Interrupting part of an asynchronous processing path

The first experiments should normally be small. Once the platform demonstrates that monitoring and recovery work as intended, the scope can be expanded.

 3. Make Observability Part of the Experiment

There is little value in breaking a system if the team cannot clearly see what happened.

The experiment should capture both technical and customer-facing signals.

Useful telemetry includes:

• Infrastructure metrics
• Application latency
• HTTP error rates
• Logs
• Distributed traces
• Pod and node health
• Queue depth
• Business transaction success rates

One of the most useful practices is correlating infrastructure events with application behavior.

A CPU spike on its own may not mean much. A CPU spike followed by increased latency and failed transactions tells a much clearer story.

 4. Validate Automated Recovery

Resilience is not just about surviving a failure. It is also about recovering from it.

Consider an EC2 instance that is intentionally terminated. The experiment should answer more than whether the instance disappeared.

The engineering team should be able to determine:

1. How quickly was the failure detected?
2. Did Auto Scaling replace the instance?
3. Did traffic continue to reach healthy capacity?
4. Did application latency increase?
5. Were customers affected?
6. How long did recovery take?

The same principle applies to Kubernetes, serverless applications, queues, databases, and downstream services.

 5. Turn Findings Into Engineering Changes

A chaos experiment is only useful when its results lead to action.

An experiment might uncover:

• An overly aggressive timeout
• A missing health check
• Insufficient pod replicas
• An ineffective scaling policy
• A dependency that has no fallback
• An alert that fires too late
• A recovery mechanism that works in theory but not under load

Those findings should feed back into architecture, automation, monitoring, and operational procedures.

That is where the real value of Chaos Engineering appears.

Challenges in Implementing Chaos Engineering

Production Risk

The biggest concern is obvious: introducing a failure can create a real outage.

That is why experiments need clearly defined boundaries. Teams typically begin in development or staging environments and move toward carefully selected production experiments only after the control mechanisms are proven.

Blast radius, experiment duration, target resources, and automatic stop conditions should all be defined before the test begins.

Choosing the Right Experiments

It is easy to create an experiment that looks impressive but provides very little useful information.

A better approach is to start with realistic failure scenarios.

Historical incidents, critical dependencies, known single points of failure, and important recovery paths are good sources for experiment ideas.

For example, if a service depends heavily on an external API, testing what happens when that API becomes slow may provide more value than simply terminating an unrelated EC2 instance.

 Incomplete Observability

An engineering team may know that something failed without knowing why users were affected.

Without sufficient telemetry, it becomes difficult to distinguish between:

Infrastructure failure → application degradation → customer impact

Observability should therefore be established before aggressive experimentation begins.

 Recovery Does Not Always Behave as Expected

A recovery mechanism can look correct in a configuration file and still fail during a real incident.

A scaling policy may react too slowly. A health check may identify problems too late. Kubernetes may recreate a pod while the underlying dependency remains unavailable.

Chaos experiments provide an opportunity to test these mechanisms under controlled conditions.

Organizational Resistance

The idea of intentionally breaking infrastructure can understandably make teams uncomfortable.

The solution is to introduce structure around the practice.

Every experiment should have:

• A clearly defined hypothesis
• An identified owner
• A limited blast radius
• Success criteria
• Abort conditions
• A recovery plan
• A documented outcome

When the objective is presented as resilience validation rather than disruption, Chaos Engineering becomes much easier to integrate into normal engineering processes.

Best Practices for Chaos Engineering

1. Start with Hypothesis

Hypothesis should be done on every experiment before starting.

2. Keep the Blast Radius Explicit

Before injecting a failure, document exactly what can be affected.

Define:

• Environment
• AWS account
• Region
• Target resources
• Maximum percentage of affected workloads
• Experiment duration
• Abort conditions

This keeps the experiment controlled even when the target system is distributed.

3. Use Automated Stop Conditions

CloudWatch alarms can act as safety mechanisms.

For example:

If application error rate exceeds the defined threshold:

CloudWatch Alarm → AWS FIS Stop Condition → Experiment Termination

This prevents an experiment from progressing into an uncontrolled outage.

4. Integrate Chaos Into the DevOps Lifecycle

Chaos Engineering works best when it becomes part of normal engineering rather than an occasional special event.

Experiments can be incorporated into:

• Release validation
• CI/CD workflows
• Disaster recovery exercises
• SRE practices
• Quarterly resilience reviews
• Major architecture changes

The level of automation can grow as the organization becomes more comfortable with the process.

5. Test Dependencies, Not Only Servers Modern applications rarely fail because a single server goes down.

Applications depend on databases, caches, queues, APIs, authentication systems, and other services.

Useful experiments can therefore include:

• Database latency
• Cache unavailability
• External API degradation
• Message queue backlog
• Network latency
• Dependency timeout
• Partial service failure

This helps validate how the application behaves when the environment around it becomes unreliable.

6. Measure Recovery as Carefully as Failure

A system failing is only half of the story. The more important question is, How quickly and safely does it recover?

• Useful measurements include:
• MTTD — Mean Time to Detect
• MTTR — Mean Time to Recover
• RTO — Recovery Time Objective
• Error rate during the experiment
• Customer-impact duration
• Automated recovery success rate

A short disruption followed by fast, predictable recovery can tell a very different story from an outage that continues for several minutes.

Chaos Engineering Lifecycle

A practical Chaos Engineering lifecycle can be summarized as:

Define → Hypothesize → Inject → Observe → Recover → Analyze → Improve

Define: Select the service, dependency, or failure scenario that should be tested.

Hypothesize: Document how the system is expected to behave.

Inject: Introduce the failure using AWS FIS or another controlled mechanism.

Observe: Monitor infrastructure, application, and business-level behavior.

Recover: Validate that automated recovery mechanisms work as expected.

Analyze: Compare the actual result with the SLO, RTO, and original hypothesis.

Improve: Fix weaknesses, update automation or architecture, and run the experiment again when appropriate.

The cycle is intentionally iterative. A good experiment often produces the next experiment.

Outcome of Implementing Chaos Engineering

A mature Chaos Engineering practice can provide several practical benefits:

• Greater confidence in resilience because recovery assumptions are tested rather than simply documented.
• Faster recovery because weaknesses in automated remediation are identified before a major incident.
• Earlier discovery of single points of failure across infrastructure and application dependencies.
• Better incident preparedness because engineers have already exercised realistic failure scenarios.
• Improved monitoring because experiments expose gaps in alerts, dashboards, and telemetry.
• Stronger disaster-recovery readiness through repeated validation of recovery paths and RTOs.
• Continuous architectural improvement because experiment findings are fed back into engineering decisions.

Conclusion: Resilience Is an Engineering Capability

High availability and resilience are related, but they are not the same thing.

Running workloads across multiple Availability Zones, configuring Auto Scaling, and adding redundant components can create a strong architecture. But those mechanisms still need to be tested under failure conditions.

Chaos Engineering provides a structured way to do that.

With AWS Fault Injection Service, Kubernetes, CloudWatch, observability platforms, automated recovery, and SLO-driven engineering, teams can move beyond assuming that systems will recover and start collecting evidence that they actually do.

More importantly, the practice creates a feedback loop between production architecture and engineering decisions. Every meaningful experiment answers a question, and every unexpected result creates an opportunity to improve the platform.

As cloud environments become more distributed, resilience cannot be treated as a one-time architecture exercise.

It has to be practiced.

The goal of Chaos Engineering is not to create chaos. It is to build confidence in what happens when the unexpected occurs.