The Disaster You're Not Ready For
Database corruption. Ransomware. Region failure. Accidental deletion. One of these will happen to you. The question is whether you recover in 30 minutes or 30 days.
Disaster recovery (DR) is not optional for SaaS. It's the difference between a bad day and bankruptcy.
RPO and RTO: The Core Metrics
RTO (Recovery Time Objective): How long until the system is back up and serving traffic?
- 1 hour RTO: You have 1 hour after failure to restore service.
- Critical services: 15-30 minutes.
- Non-critical services: 4-24 hours.
RPO (Recovery Point Objective): How much data are you willing to lose?
- 1-hour RPO: You can lose up to 1 hour of data.
- 15-minute RPO: Stricter; requires more frequent backups.
- Zero RPO: No data loss (requires synchronous replication; expensive).
Your SLA should tie to these. Communicate to customers: "We guarantee 99.9% uptime with 1-hour RTO."
Backup Strategy
Automated daily backups: Every database, every 24 hours, backed up to a separate storage system (S3, another region, different cloud provider).
Incremental backups: First backup is full (2TB). Subsequent backups only save changes (+10GB daily). Faster and cheaper.
Point-in-time recovery: Retain 30-90 days of backups. If corruption happens at 3pm, restore to 2:50pm.
Backup testing: Restore a backup weekly to ensure it works. Nothing is worse than discovering at 2am that backups are corrupted.
# Weekly backup restore test
RESTORED_DB=$(restore_backup_to_staging())
run_queries_on($RESTORED_DB) # Verify data integrity
drop_database($RESTORED_DB) # Clean up
Multi-Region Failover
Single-region deployments fail. A regional outage (AWS us-east-1 goes down, it has happened) makes you unreachable.
Active-passive: Primary region serves traffic. Standby region is idle. On failure, DNS points to standby. Data is replicated continuously.
- Advantages: Simple, cost-effective.
- Disadvantages: Standby resources can drift; RTO is 5-15 minutes (DNS propagation, failover automation).
Active-active: Both regions serve traffic simultaneously. Load balanced 50/50.
- Advantages: Better RTO, no resource waste.
- Disadvantages: Complex (distributed transactions, eventual consistency), higher cost (2x infrastructure).
Start with active-passive. Add active-active only if 1-hour RTO is unacceptable.
Failover Automation
Manual failover is slow and error-prone. Automate it.
If primary region health check fails for 2 minutes:
1. Run sanity checks on standby region
2. Promote standby database to primary
3. Update DNS to point to standby
4. Alert on-call team
5. Begin investigation
Tools: Terraform, Kubernetes operators, or cloud provider failover services (AWS RDS with multi-AZ, GCP Cloud SQL with HA).
Testing Your Disaster Plan
A plan untested is a plan that won't work. Run disaster drills quarterly.
Drill procedure:
- Kill the primary database. (Seriously.)
- Set a timer. Measure how long failover takes.
- Have the team debug. This is learning.
- Restore the primary. Return to normal.
Target: failover and full service recovery in under your RTO.
Real example: Company A never tested failover. Their RTO was "1 hour." When a real outage hit, it took 4 hours to realize backups were corrupt, find a good backup, restore, and verify. Customers lost 4 hours of data. The company lost significant contracts.
Company B tested monthly. When the same outage hit them, they failed over in 12 minutes, with zero data loss.
Database Replication
For databases, continuous replication is essential. Real-time standby databases receive writes milliseconds after the primary.
Options:
- RDS Multi-AZ: AWS handles replication automatically. Higher cost, zero complexity.
- Streaming replication: PostgreSQL with wal_level=replica replicates log entries in real-time to standby.
- MySQL Group Replication: Multiple replicas with automatic failover.
For SaaS, managed solutions (RDS Multi-AZ, Cloud SQL HA) are worth the cost. One less thing to maintain.
Document and Practice
DR only works if your team knows the plan.
Create a runbook:
DISASTER RECOVERY RUNBOOK
=========================
Primary database is down:
1. Check AWS health dashboard for regional issues
2. SSH to standby-db-01
3. Run: sudo systemctl status postgres
4. If postgres is running but slow, check disk space
5. If standby is healthy, update DNS to failover IP
6. Update PagerDuty with new status
7. Start the debugging war room
Contact: Engineering lead on-call
Practice annually: Full team participates. Act like it's 3am and you're groggy. This reveals missing steps.
Checklist for DR Readiness
- Define RPO and RTO (documented and communicated to customers)
- Automated daily backups to a separate storage location
- Weekly backup restore tests
- Standby infrastructure (passive or active-active)
- Failover automation (not manual)
- Monitoring and alerting for failure detection
- Quarterly failover drills
- Runbook documentation
- Team training and cross-training
Disaster recovery isn't fun. But recovering from a disaster after you've prepared is infinitely better than improvising at 3am with customers calling.
Frequently asked questions
What's a realistic RTO for most SaaS startups?
1-4 hours for active-passive, 15-30 minutes for active-active with automation. If customers need less than 1-hour RTO, you need active-active or expensive redundant infrastructure. Most startups target 1-hour RTO initially.
How much does multi-region DR cost?
Active-passive: roughly 60-70% of primary region cost (standby is smaller). Active-active: 200% of single-region cost (2x resources). For critical services, this is essential. For non-critical, save the cost and accept longer RTO.
Should we replicate to multiple clouds (AWS + GCP + Azure)?
Rarely cost-effective for startups. Multi-cloud adds operational complexity and vendor negotiation overhead that exceeds benefits. Focus on multi-region within one cloud first. Multi-cloud is insurance for vendor pricing/terms changes, not primary DR.