Canary Releases

What are canary releases?

A canary release is a deployment strategy where a new version is gradually rolled out to a small percentage of users first, with metrics monitored before expanding to full rollout. If errors are detected, the canary is rolled back before most users are affected.

The name comes from the historical practice of bringing canary birds into coal mines - if the canary died, miners knew to evacuate before being overcome themselves.

How It Works

Release timeline:
Day 1: 1% of traffic -> new version (canary)
Day 2: 10% -> monitor metrics
Day 3: 50% -> monitor metrics
Day 4: 100% -> old version decommissioned

If error rate increases at any step:
  -> Roll back: 0% to new version, 100% to old

Traffic Splitting Mechanisms

Weighted Routing at Load Balancer

• Configure load balancer to send X% of requests to new version servers
• AWS ALB: weighted target groups
• Nginx: upstream with weight directive
• Random distribution - users may experience different versions on different requests

Feature Flag-Based

• Deploy new code to all servers
• Feature flag controls which users see the new behaviour
• Route by user ID, cohort, or percentage
• Same user consistently gets same version (consistent experience)

Kubernetes Rolling Update

Kubernetes rolling updates are a form of canary: gradually replace old pods with new pods.

strategy:
  type: RollingUpdate
  rollingUpdate:
    maxSurge: 1        # max extra pods during update
    maxUnavailable: 0  # min available pods during update

With 10 replicas: starts replacing 1 pod at a time. During transition, ~90% serve old version, ~10% serve new.

Metrics-Based Promotion

Automate the decision to expand or roll back based on metrics:

If error rate > baseline + 0.1%: roll back
If p99 latency > baseline + 50ms: pause, investigate
If both healthy for 30 minutes: expand to next percentage

Tools: Argo Rollouts (Kubernetes), Spinnaker, AWS CodeDeploy with alarms, Flagger.

Automatic Rollback Triggers

Define rollback conditions upfront:

• HTTP 5xx error rate > 1%
• p95 latency > 500ms
• Business metric: checkout completion rate drops > 2%

Monitor for at least 15-30 minutes at each percentage (slow-burning bugs may not appear immediately under low traffic).

Canary vs Blue-Green

	Blue-Green	Canary
Traffic split	0% then 100%	Gradual
Risk exposure	100% of users at once	Small % initially
Rollback speed	Instant	Fast
Cost	2× infra during deployment	Marginal extra pods
Database compatibility	Harder (both versions active briefly)	Harder (both versions active longer)

Canary + Feature Flags

Combining both gives maximum control:

• Deploy new code to 100% of servers (no user impact yet)
• Enable feature flag for 1% of users
• Expand flag rollout gradually based on metrics
• Decouple deploy from release entirely

Interview Angle

For "how do you release features safely at scale?", canary + automated rollback is the senior answer. Show you understand: traffic splitting mechanism, metrics to monitor, automatic rollback criteria, and the database compatibility issue that affects any strategy where two versions run simultaneously.