Building Self-Healing Infrastructure

Infrastructure That Heals Itself

A server crashes. Instead of waiting for an alert and manual restart, the system automatically:

1. Detects the crash (health check)
2. Removes failed server from load balancer
3. Starts a new server
4. Routes traffic to the new server
5. Alerts the team

Uptime improves. Oncall burden decreases. This is self-healing infrastructure.

Health Checks: Detection

Health checks run continuously. Services report: "I'm healthy" or "I'm sick."

HTTP health checks:

GET /health → 200 OK {"status": "healthy"}
GET /health → 503 Service Unavailable {"status": "database error"}

If 3 consecutive checks fail, mark server as unhealthy.

Implementation:

from flask import Flask, jsonify
import psycopg2

app = Flask(__name__)

@app.route('/health')
def health():
  try:
    # Check database connectivity
    conn = psycopg2.connect("dbname=prod user=app")
    conn.close()
    return jsonify({"status": "healthy"}), 200
  except:
    return jsonify({"status": "database unavailable"}), 503

What to check:

• Database connectivity
• External API availability
• Disk space
• Memory usage
• Application logic (can we handle requests?)

A health check that only returns 200 is useless. Check dependencies.

Auto-Scaling: Adding Capacity

When demand increases, spin up new servers. When demand decreases, shut them down.

Kubernetes auto-scaling:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: myapp-scaler
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: myapp
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70

When CPU > 70%, add more pods (up to 10 max). When CPU < 30%, remove pods (down to 2 min).

AWS Auto Scaling: Similar concept. Launch new EC2 instances when load increases.

Caveats:

• Scale time: if it takes 5 minutes to launch a server, you have 5 minutes of overload.
• Cost: more servers = more cost. Set caps.
• Database bottleneck: scaling app servers doesn't help if database is the bottleneck.

Circuit Breakers: Fail Fast

If an external API is slow, don't wait for timeout. Fail fast, let traffic bypass the slow service.

import circuitbreaker

@circuitbreaker.circuit(failure_threshold=5, recovery_timeout=60)
def call_external_api():
  response = requests.get('https://api.external.com/data', timeout=5)
  return response.json()

# After 5 failures in 60 seconds, circuit opens
# Further calls immediately return error without hitting the API
# After 60 seconds, circuit half-opens, test if API recovered
# If test succeeds, circuit closes (resume normal operation)

Circuit breaker states:

• Closed: normal operation, calls go through
• Open: too many failures, calls immediately fail
• Half-open: testing if service recovered

Automated Failover

If a server fails, automatically switch to a standby.

Master-standby database:

Primary DB (active)
   ↓
Standby DB (standby, real-time replication)

If primary fails:
  1. Promote standby to primary
  2. Update DNS/connection strings
  3. Traffic flows to new primary
  4. Repair old primary, make it new standby

Multi-zone deployment:

Zone A: App server + database
Zone B: App server + database (standby)

If Zone A fails:
  1. Health checks detect failure
  2. Load balancer removes Zone A
  3. All traffic routes to Zone B
  4. Repair Zone A

Automation: define this in code (Infrastructure as Code), test regularly.

Chaos Engineering: Test Your Resilience

Intentionally break things in production (carefully) to test auto-recovery.

Chaos experiments:

• Kill a random pod: does the system restart it?
• Inject latency into a service: does circuit breaker activate?
• Fill a disk to 90%: does the app handle it gracefully?
• Disconnect a region: does failover work?

Tools: Gremlin, Chaos Toolkit, Kubernetes chaos experiments.

Process:

1. Define baseline (normal operation metrics)
2. Run chaos experiment (kill a pod)
3. Measure impact (does error rate increase? by how much?)
4. Fix issues if needed
5. Re-run to verify fix

Schedule chaos experiments weekly. Find problems before customers do.

Monitoring for Auto-Recovery

For auto-recovery to work, you need visibility.

Metrics to monitor:

• Health check failure rate
• Server restarts (should be rare, frequent restarts indicate problems)
• Auto-scaling events (too frequent = bad tuning)
• Database failover events (should not happen often)

Alerting:

alert: HighServerRestartRate
if: rate(server_restarts[5m]) > 0.1  # More than 1 restart per 5 minutes
action: page on-call engineer

Frequent restarts indicate a problem that auto-recovery masks. Investigate root cause.

Self-Healing Checklist

• Health checks on all services (check dependencies, not just "alive")
• Auto-scaling configured (min/max replicas, scaling trigger)
• Circuit breakers for external API calls
• Automated failover for databases and critical services
• Monitoring and alerting on failures
• Chaos experiments to test auto-recovery
• Runbook for manual intervention (for situations auto-recovery can't handle)
• Testing of auto-recovery monthly
• Quick reprovision time (can you rebuild a server in <5 minutes?)