Webhook Architecture: Reliable Event Delivery

The Webhook Problem

Your SaaS sends webhooks to customers. "Order paid. Here's the data."

Customer's server is down for 30 seconds. Your webhook times out. You don't retry. Customer's system doesn't know about the payment. Money is lost.

Or worse: your webhook retries 5 times. Customer's server receives it 5 times. They process the payment 5 times. Nightmare.

Reliable webhooks require careful design.

Webhook Delivery Guarantees

Three options:

At-most-once: Send once, don't retry. If delivery fails, tough luck.

• Simplest to implement.
• Data loss risk.
• Use only for non-critical events (analytics, logs).

At-least-once: Send, retry on failure. Guaranteed delivery, but may deliver twice.

• Customer must handle duplicates (idempotency).
• Realistic for most use cases.

Exactly-once: Deliver exactly once, no more, no less.

• Theoretically impossible in distributed systems (Google Cloud docs admit this).
• Practically: use at-least-once + idempotency.

Most SaaS uses at-least-once. Customers build idempotency.

Retry Logic: Exponential Backoff

Don't retry immediately. Retry with exponential backoff:

Attempt 1: immediately (0s delay)
Attempt 2: after 2 seconds
Attempt 3: after 4 seconds
Attempt 4: after 8 seconds
Attempt 5: after 16 seconds
Attempt 6: after 32 seconds
... max wait: 1 hour

After 6 retries (total 63 seconds elapsed), give up. Send an alert to your team.

Why exponential backoff:

• If customer's server is rebooting, immediate retry fails.
• Backing off gives time to recover.
• Exponential growth prevents hammering a struggling server.
• Cap at reasonable max (1 hour) so you don't wait forever.

Implementation:

import time

def send_webhook_with_retry(url, data, max_retries=6):
  for attempt in range(max_retries):
    try:
      response = requests.post(url, json=data, timeout=10)
      if response.status_code == 200:
        return True  # Success
      elif response.status_code >= 500:
        # Server error, retry
        raise Exception(f"Server error: {response.status_code}")
      else:
        # Client error, don't retry
        log_webhook_failure(url, data, response.status_code)
        return False
    except Exception as e:
      if attempt == max_retries - 1:
        # Last attempt, give up
        alert_team(f"Webhook to {url} failed after {max_retries} retries")
        return False

      # Exponential backoff
      wait_time = 2 ** attempt
      time.sleep(wait_time)

  return False

Idempotency: Handle Duplicates

Customers must handle duplicate deliveries. Provide an idempotency key.

Webhook payload:

{
  "event_id": "evt_abc123def456",
  "event_type": "order.paid",
  "timestamp": 1706876400,
  "data": {
    "order_id": "ord_789",
    "amount": 9999
  }
}

Customer saves event_id. If they receive the same event_id again, they ignore it.

# Customer-side code
received_event_id = request.json['event_id']

if database.has_event(received_event_id):
  return 200  # Already processed, return success

# Process the event
database.record_payment(request.json['data'])
database.save_event(received_event_id)

return 200

Webhook Security

Signature verification: Prove the webhook came from you, not an imposter.

Use HMAC. Generate a signature with a shared secret:

import hmac
import hashlib
import json

# On your side (sending webhook)
secret = "your_shared_secret"
payload = json.dumps({"order_id": "123", "amount": 9999})
signature = hmac.new(secret.encode(), payload.encode(), hashlib.sha256).hexdigest()

# Send signature in header
headers = {"X-Webhook-Signature": signature}
requests.post(url, data=payload, headers=headers)

# On customer side (receiving webhook)
received_signature = request.headers.get("X-Webhook-Signature")
expected_signature = hmac.new(secret.encode(), request.data, hashlib.sha256).hexdigest()

if not hmac.compare_digest(received_signature, expected_signature):
  return 403  # Unauthorized, reject webhook

# Process webhook

HTTPS only: Webhooks must use HTTPS, never HTTP. Encrypt in transit.

Timeout: Set reasonable timeouts (5-10 seconds). Don't wait forever for slow customers.

Rate limiting: Don't send too many webhooks to one customer. Queue them.

Webhook Delivery Dashboard

Let customers see webhook delivery status:

• List of sent webhooks (event type, timestamp, status)
• Manual retry button (user can retry failed deliveries)
• Logs (what was the response? why did it fail?)
• Health status (has this endpoint been healthy recently?)

This transparency builds trust and helps customers debug integration issues.

Monitoring Webhooks

Alert on:

• High failure rate (>5% of webhooks failing)
• High latency (average delivery > 5 seconds)
• Critical endpoints with repeated failures

Track:

• Delivery success rate (percentage of webhooks delivered successfully)
• Delivery latency (p50, p95, p99)
• Retry counts (how many retries needed?)

alert: HighWebhookFailureRate
if: rate(webhook_failures[5m]) > 0.05
for: 10m
action: page on-call team

Webhook Checklist

• Use at-least-once delivery with exponential backoff
• Provide event IDs for idempotency
• Sign webhooks with HMAC
• HTTPS only (no HTTP)
• Reasonable timeout (5-10 seconds)
• Retry logic capped at reasonable max (6 retries, 1 hour max wait)
• Queue webhooks to avoid rate limiting
• Webhook delivery dashboard for customers
• Monitoring and alerting for high failure rates
• Documentation for webhook security