Design a notification service that sends 50 million emails, SMS and push notifications per day with user preferences and retries.
Short answer
Accept notification requests via API/events into durable queues per channel and priority, apply user preferences, templates and rate limits, send through providers with retries, failover and idempotency keys, track delivery status from provider callbacks, and isolate slow providers with separate worker pools.
Detailed answer
- Ingestion: API and event consumers create notification jobs with idempotency keys (
userId + templateId + businessEventId). - Processing: preference service (opt-outs, quiet hours, channel choice), template rendering with localization, deduplication.
- Queues per channel and priority (OTPs and security alerts high priority; marketing low) so bulk campaigns never delay OTPs.
- Senders: per-provider workers with rate limits matching provider quotas, retries with backoff, circuit breakers, failover to a secondary provider.
- Status tracking: provider webhooks update delivery status; analytics events.
- Scale: 50M/day ≈ 580/s average, campaigns spike far higher; queues absorb spikes.
Architecture diagram
flowchart LR SRC[Services / events] --> API[Notification API] API --> PREF[Preferences & templates] PREF --> QH[[High-priority queue]] PREF --> QL[[Bulk queue]] QH & QL --> EM[Email workers] --> SES[Email providers] QH & QL --> SMS[SMS workers] --> SMSP[SMS providers] QH & QL --> PUSH[Push workers] --> FCM[APNs / FCM] SES & SMSP & FCM -- webhooks --> STAT[(Delivery status)]
Follow-up questions
- How do you prevent sending the same OTP twice? (Idempotency keys; dedupe window.)
- How do you comply with regulations (e.g. DND lists, unsubscribe)? (Central preference and consent checks.)
What the interviewer expects
Priority isolation, idempotency, provider rate limits and failover, preference handling.
Common mistakes to avoid
One queue for everything, synchronous sending in the caller's request, no deduplication.