Design a Notification Service

Design a notification service that can deliver push notifications, SMS, and emails at scale. Notifications may be triggered by system events (payment confirmed) or marketing campaigns (10M users simultaneously). Delivery must be tracked.

Capacity Estimation

• Rate: 100M notifications/day = 1,157 notifications/sec average
• Peak (marketing blast to 10M users): 10M / 60 seconds = 166K/sec — must queue and fan out
• Delivery records: 100M × 365 × 3 bytes status = ~110 GB/year
• Templates: small, fit in cache

Architecture

[Event Sources]
  Payment Service → payment.confirmed event
  Marketing Tool → campaign.triggered event
           │
           ▼
    [Notification API Service]
      - validates request
      - checks user preferences
      - resolves template
      - publishes to channel-specific Kafka topic

           ↓
  ┌─────────────────────────────────────┐
  │         Kafka Topics                │
  │  notifications.push                 │
  │  notifications.email                │
  │  notifications.sms                  │
  └─────────────────────────────────────┘
           ↓
  [Channel Workers] (scale independently)
    Push Worker → APNs / FCM
    Email Worker → SendGrid
    SMS Worker → Twilio
           ↓
  [Delivery Tracker] → updates delivery_status in DB

API Design

// Trigger single notification
POST /api/v1/notifications
Body:
{
  "userId": "user-123",
  "template": "payment_confirmed",
  "data": { "amount": "$49.99", "orderId": "ORD-789" },
  "channels": ["PUSH", "EMAIL"],
  "priority": "HIGH"
}

// Trigger campaign (batch)
POST /api/v1/campaigns
Body:
{
  "segmentId": "premium_users",
  "template": "new_feature_announcement",
  "scheduledAt": "2025-09-20T09:00:00Z"
}

// Check delivery status
GET /api/v1/notifications/{notificationId}/status

Database Design

Notifications (Cassandra — write-heavy delivery logs)

notification_log
  notification_id  UUID    PK
  user_id          UUID
  channel          ENUM    (PUSH, EMAIL, SMS)
  template_id      VARCHAR
  status           ENUM    (PENDING, SENT, DELIVERED, FAILED, CLICKED)
  created_at       TIMESTAMP
  delivered_at     TIMESTAMP

User preferences (PostgreSQL)

notification_prefs
  user_id          UUID    PK
  channel          VARCHAR
  enabled          BOOLEAN
  updated_at       TIMESTAMP

Templates (Redis cache backed by DB)

template:{templateId}  →  { subject, body_html, body_text }

Scaling for Campaign Blasts

Sending to 10M users in <60 seconds requires:

1. Pre-compute recipient lists (async, before scheduled time)
2. Parallel Kafka partitions: 100 partitions × 100K messages/partition = 10M messages fanned out
3. Worker fleet: 100 push workers × 1,600 notifications/sec each = 160K push/sec
4. APNs/FCM limits: Both support batch send APIs (~1000 tokens per batch call)

Campaign trigger → segment resolver → writes 10M user IDs to Kafka
                                              │
                               100 Kafka partitions
                                              │
                               100 Push Worker instances
                                              │
                               10M APNs/FCM calls over 60 seconds

• At-least-once vs exactly-once: Kafka guarantees at-least-once. De-duplicate at the client (track notification_id) to avoid showing the same notification twice.
• Fire-and-forget vs delivery tracking: tracking requires storing delivery records (~100M rows/day) — adds cost and latency. Make it opt-in per notification type.
• Push vs in-app notification: push reaches offline users but can't be revoked; in-app is revocable but requires an open session.

• APNs/FCM rate limits: enforce per-app-per-token limits by distributing across multiple APNs connections
• Invalid tokens: ~5% of push tokens are stale. Purge invalid tokens from DB based on APNs/FCM feedback service
• Email reputation: high bounce/spam rates lower sender score. Use dedicated IPs and honour unsubscribes immediately

• APNs down: messages buffered in Kafka. Workers retry with exponential backoff. Campaign delayed, not lost.
• SMS provider failure: fallback to secondary SMS provider (Vonage → Twilio).
• Worker crash: Kafka consumer group rebalances — unprocessed messages reassigned to healthy workers.