Why REST APIs Break at Scale
Synchronous REST requests create coupling. When service A calls service B, A waits. If B is slow, A blocks. If B is down, A fails. Users see cascading outages.
At scale, this becomes untenable. A payment service calls an email service calls a logging service calls a metrics service. One slow link stalls the entire chain.
Event-driven architectures solve this: services emit events, not API calls. Other services react asynchronously. If a consumer is slow or down, producers don't care. Resilience improves. Scaling becomes independent.
Message Queue Types
RabbitMQ: Traditional message broker. Reliable delivery, rich routing, proven. Excellent for work queues and pub/sub. Requires operational overhead.
Apache Kafka: Distributed event streaming. Partitions data, allows replay, higher throughput than RabbitMQ. Better for event sourcing and analytics pipelines. Heavier operational footprint.
AWS SQS/SNS: Managed cloud services. Zero operational overhead. Less feature-rich than RabbitMQ. Pay per message (expensive at scale, cheap at startup).
For startups, SQS is often the right choice: managed, scales automatically, cheap initially.
Publish-Subscribe vs Work Queues
Pub/Sub (Broadcast): One producer emits an event; multiple consumers listen. Each gets a copy.
- Use for: Order placed → email notification, inventory update, analytics, audit log.
- Tool: Kafka, RabbitMQ with fanout exchanges, SNS.
Work Queue (Task Distribution): One producer, one consumer per task. Tasks are distributed.
- Use for: Video encoding, report generation, payment processing.
- Tool: SQS, RabbitMQ with direct exchanges, Celery.
Event-Driven Order Service
Example architecture:
Order Service receives POST /orders
↓
Saves order to database
↓
Emits OrderCreated event to Kafka
↓
Payment Service consumes, charges card
↓
Emits PaymentProcessed event
↓
Inventory Service consumes, reserves stock
↓
Email Service consumes, sends confirmation
↓
Analytics Service consumes, tracks metrics
Order Service doesn't wait for payment, email, or inventory. They process asynchronously. If email is slow, orders don't back up. If inventory is down, payment still completes.
Handling Failures
Asynchronous failures are invisible if not handled. A message consumed but not processed isn't a problem; it's a silent outage.
Dead-letter queues (DLQs): Messages that fail processing after N retries go to a DLQ for manual review.
Consumer processes message
→ If it fails, retry (exponential backoff)
→ After 3 retries, send to DLQ
→ Operator is alerted, investigates, reprocesses
Idempotency: Consumers must handle duplicate messages. If a message is delivered twice, the outcome should be identical.
if (database.hasOrderId(event.orderId)) {
// Already processed, ignore
return;
}
// Process for first time
Ordering Guarantees
Kafka guarantees ordering within a partition. If you emit events A, B, C to the same partition key, consumers see them in order.
RabbitMQ doesn't guarantee ordering across multiple consumers, but does within a single queue.
For operations where order matters (state machines, ledgers), use Kafka with keyed events. For eventual consistency (caches, analytics), ordering is less critical.
Observability for Event-Driven Systems
Debugging is harder with events. You can't see the full flow in a single transaction log.
Strategies:
- Correlation IDs: Attach a trace ID to events. Follow the ID across services.
- Event sourcing: Log every event. Rebuild state by replaying events.
- Distributed tracing: Use Jaeger or Zipkin. Trace follows events across services.
Without observability, event-driven systems become black boxes.
Choosing Your First Technology
Startup recommendation: Start with SQS. Managed, simple, no ops overhead. As volume grows and features (ordering, replay) become essential, migrate to Kafka.
Build your architecture assuming migration is possible: abstract the queue behind an interface. Swap SQS for Kafka later without rewriting application code.
Frequently asked questions
Should we use events for all communication?
No. Use synchronous REST/gRPC for request-response patterns where you need immediate feedback (login, payment validation). Use events for notifications and state changes that don't require immediate response (email, analytics, cache invalidation).
How do we ensure exactly-once message processing?
True exactly-once is impossible in distributed systems. Use idempotency instead: design consumers so processing twice has the same effect as once. Pair with deduplication IDs and transactional outbox pattern for guaranteed delivery.
What's the operational overhead of Kafka vs RabbitMQ vs SQS?
SQS: nearly zero, it's managed. RabbitMQ: moderate, requires monitoring and tuning. Kafka: high, requires cluster management, monitoring, and deep operational expertise. For teams with <10 people, SQS is wise; RabbitMQ if you need more features and can afford ops time.