Monitoring and Observability for Production Systems

Monitoring vs Observability

Monitoring tells you what's broken. Observability tells you why.

Monitoring: "CPU is at 95%." Observability: "CPU is high because request latency spiked due to a full table scan in the payments query, which happened because we deployed a change that removed an index."

Monitoring is backwards-looking and limited. Observability is forward-looking and exploratory.

The Three Pillars

Metrics: Quantitative data. CPU, memory, request rate, error rate, latency. Aggregated over time.

Logs: Events. "User 123 logged in," "payment failed with timeout," "database query took 2.5s."

Traces: Full request paths. User clicks button → frontend API call → backend service → database query. View the entire journey.

Metrics with Prometheus

Prometheus scrapes metrics from services every 15 seconds. Services expose endpoints returning metrics in a simple text format.

http_requests_total{method="GET",status="200"} 1234
http_requests_total{method="POST",status="500"} 2
http_request_duration_seconds{le="0.1"} 100
http_request_duration_seconds{le="1"} 250

Key metrics to track:

• Request rate (requests per second)
• Error rate (5xx responses as % of total)
• Latency (p50, p95, p99 response time)
• Saturation (CPU, memory, disk, database connections)

Alerting on metrics:

alert: HighErrorRate
if: rate(http_requests_total{status=~"5.."}[5m]) > 0.05
for: 5m
then: page oncall engineer

If error rate exceeds 5% for 5 minutes, alert. This is threshold-based alerting. Simple and effective for known-bad scenarios.

Structured Logging

Logs aren't grep-friendly text. Structure them as JSON:

{
  "timestamp": "2025-09-15T14:23:45Z",
  "level": "error",
  "service": "payment-service",
  "userId": "usr_123",
  "orderId": "ord_456",
  "error": "payment_gateway_timeout",
  "retries": 2,
  "duration_ms": 5000
}

Parse logs into a tool like ELK Stack, Datadog, or Loki. Query by any field:

service:payment-service AND error:payment_gateway_timeout AND retries:2

Structured logs make debugging fast. Unstructured text logs are archaeology.

Distributed Tracing

In microservices, a single user action touches multiple services. Tracing follows the full path.

User places order → Order Service → Payment Service → Inventory Service → Notification Service

A trace assigns one ID to the entire flow. Each service adds span data:

• Service name
• Operation name
• Latency
• Success/failure
• Custom attributes

Tools: Jaeger, Zipkin, or cloud-native offerings (Datadog, AWS X-Ray).

Traces answer: "Why is checkout slow?" You see the order service took 100ms, payment took 2s, inventory took 50ms. Payment is the bottleneck. Look at payment service logs with the trace ID, find why it's slow.

Building Observability Into Code

Instrumentation: Export metrics and logs from your application.

For Go/Python/Node:

from prometheus_client import Counter, Histogram

request_count = Counter('requests_total', 'Total requests')
request_duration = Histogram('request_duration_seconds', 'Request latency')

@app.route('/api/checkout')
@request_duration.time()
def checkout():
  request_count.inc()
  # ... checkout logic

Logging: Use structured logging libraries.

logger.info("checkout_started", extra={
  "userId": user_id,
  "cartValue": cart_value,
  "items": len(cart)
})

Tracing: Add trace context propagation. Pass a trace ID through services.

trace_id = request.headers.get('X-Trace-ID')
logger.info("processing_payment", extra={"trace_id": trace_id})
call_payment_service(headers={'X-Trace-ID': trace_id})

Alert Design

Alerts wake up on-call engineers at 3am. Bad alerts cause burnout.

Good alert: Actionable, low false positive rate, points to solution.

• "Database connection pool exhausted" → increase pool size or find connection leak.

Bad alert: High false positive rate, requires investigation.

• "CPU above 60%" → CPU oscillates; this fires and clears constantly.

Use SLOs and error budgets:

• SLO: "99.9% of requests succeed within 500ms"
• Alert when you'll miss SLO at current error rate
• Ignore temporary spikes; alert on sustained degradation

Observability Stack Recommendation

Startup phase (< $500/month):

• Prometheus + Grafana for metrics
• Loki for logs
• Jaeger for traces
• All open-source, self-hosted on a single instance

Growth phase (< $5k/month):

• Datadog or New Relic for integrated platform
• Trade DIY for managed experience and better UX

Scale phase:

• Dedicated observability infrastructure
• Consider multi-vendor (Prometheus + Datadog + Jaeger)