WhatsApp-like Messaging Platform

Real-time messaging for hundreds of millions of devices with persistent connections, delivery receipts, offline queues, groups, media, push notifications and end-to-end encryption.

[!FACT] Public context WhatsApp has publicly described running its chat backend on Erlang/FreeBSD with a small engineering team, and uses the Signal Protocol for end-to-end encryption (announced in 2016). Exact current internals are not public.

[!ASSUMPTION] Scope "Chirp", a hypothetical messenger with 500 M daily active users. Numbers are design assumptions.

1. Business problem and objectives

Build a reliable, private, low-latency messenger that works on poor mobile networks. Objectives:

  • Message delivery p95 < 300 ms when both users are online in the same region.
  • No message loss once the server acknowledges it; in-order delivery per conversation.
  • End-to-end encryption by default: the server cannot read content.
  • Work on 2G/3G networks and low-end Android devices; minimal battery and data use.
  • Groups up to 1,024 members; media sharing; multi-device support.

2. Business model and revenue generation

Stream Notes
Business messaging API Businesses pay per conversation/message for customer notifications and support
Click-to-chat ads Ads on other properties open chats with businesses
Payments in chat Transaction fees or ecosystem value
Premium business tools Multi-agent inboxes, catalogs

The consumer app is free; architecture efficiency (cost per user) is critical.

3. Functional requirements

  1. One-to-one messaging with sent/delivered/read receipts.
  2. Group messaging with admin controls.
  3. Offline delivery: messages queued until the recipient connects.
  4. Media (images, video, voice notes, documents) with thumbnails.
  5. Presence and typing indicators (best effort).
  6. Push notifications when the app is backgrounded.
  7. Multi-device (phone + desktop/web) with E2EE.
  8. Contact discovery, blocking, reporting abuse.
  9. Business accounts and API.

4. Non-functional requirements

Attribute Target
Latency p95 < 300 ms same-region online delivery
Availability 99.99% for send/receive
Durability No loss after server ack; undelivered messages retained up to 30 days
Ordering Per-conversation ordering
Scale 500 M DAU, 200 M concurrent connections at peak, 100 B messages/day
Privacy E2EE; minimal metadata retention
Efficiency Low bandwidth protocol, battery-aware

5. Expected user base and traffic assumptions

  • 100 B messages/day ≈ 1.16 M messages/s average, ~3 M/s peak (New Year's Eve can spike far above).
  • Average encrypted text message ~200 bytes + headers; group fan-out multiplies deliveries (average group message delivered to ~10 devices).
  • 200 M concurrent persistent connections; if one gateway node holds ~1 M connections (possible with tuned Erlang/epoll stacks), ~200+ nodes plus headroom.
  • Media: 7 B media items/day, average 200 KB → ~1.4 PB/day ingress, served via CDN; retained only until downloaded plus a window (design choice).

6. High-level architecture

  • Connection gateways: hold persistent connections (custom binary protocol over TCP/TLS or WebSocket/MQTT-like), authenticate devices, route messages.
  • Session registry: which gateway node holds each device's connection.
  • Message router: delivers to the recipient's gateway if online, else stores in the offline queue.
  • Message store: durable per-recipient (or per-conversation) queue until acknowledged.
  • Group service: membership and fan-out.
  • Media service: upload/download of encrypted blobs via object storage and CDN.
  • Push service: APNs/FCM notifications for offline/background devices.
  • Key directory: public identity keys and prekeys for E2EE session setup.

7. Detailed component architecture

Component Design choice Why
Gateway Erlang/Elixir or Go/Rust with epoll; one lightweight process/goroutine per connection Millions of mostly idle connections per node
Session registry Distributed in-memory store (sharded by user ID), with TTL heartbeats Fast lookups; rebuilt on reconnect
Router Stateless; consistent hashing on recipient ID Even load, local decisions
Offline store Wide-column store keyed by (recipient_device, message_seq) High write rate, append-heavy, delete on ack
Group service Membership store + fan-out workers Large groups fan out asynchronously
Media Pre-signed upload to object storage; encrypted blob; CDN for download Keeps heavy bytes off chat servers
Push Queue-backed workers to APNs/FCM Provider rate limits and failures isolated
Key directory Strongly consistent store for identity keys; prekey bundles consumed once E2EE correctness

8. Architecture diagram

flowchart LR
  A[Alice device] <-->|persistent TLS conn| G1[Gateway node 17]
  B[Bob device] <-->|persistent TLS conn| G2[Gateway node 42]
  G1 --> R[Message router]
  R --> SR[(Session registry<br/>user → gateway)]
  R -->|online| G2
  R -->|offline| OQ[(Offline message store<br/>per device)]
  R --> GRP[Group fan-out service]
  GRP --> R
  OQ -->|on reconnect| G2
  R -->|Bob backgrounded| PUSH[Push service] --> APNS[APNs / FCM]
  A -->|encrypted media| MED[Media upload<br/>pre-signed URL] --> OBJ[(Object storage)] --> CDN[CDN]
  CDN --> B
  KD[(Key directory<br/>identity + prekeys)] --- A
  KD --- B

9. Frontend architecture

  • Native apps (Kotlin, Swift) for performance and battery; desktop via Electron or native; web via WebSocket.
  • Local encrypted database (SQLite/SQLCipher) is the source of truth for chat history on the device.
  • Outbox on device: messages queued locally with client-generated message IDs; retried until server ack.
  • Adaptive media quality by network; background download policies.
  • Signal Protocol libraries for E2EE; key verification (safety numbers/QR).

10. Backend architecture

  • Protocol: compact binary frames; client message ID for dedup; server assigns per-conversation sequence for ordering.
  • Delivery semantics: at-least-once from client to server (retries with same client ID; server dedups), at-least-once server to device with device ack; device dedups by message ID. Effective exactly-once display.
  • Receipts are messages themselves (delivered/read), flowing back to the sender.
  • Gateways are thin; business logic (groups, spam checks on metadata, rate limits) in backend services.
sequenceDiagram
  participant A as Alice
  participant GA as Gateway A
  participant RT as Router
  participant ST as Offline store
  participant GB as Gateway B
  participant B as Bob
  A->>GA: SEND {clientMsgId, to: Bob, ciphertext}
  GA->>RT: route
  RT->>ST: persist for Bob's devices (durable)
  RT-->>GA: ACK (server accepted)
  GA-->>A: ✓ sent
  RT->>GB: deliver (Bob online)
  GB->>B: MESSAGE
  B-->>GB: ACK
  GB->>ST: delete delivered message
  GB->>RT: DELIVERED receipt to Alice
  RT->>GA: receipt
  GA-->>A: ✓✓ delivered

11. API design with sample endpoints

The chat path uses the persistent connection protocol, not REST. Control APIs:

POST /v1/devices/register           { phone, otpProof, identityKey, signedPreKey, oneTimePreKeys[] }
GET  /v1/keys/{userId}/devices      -> prekey bundles for session setup (one-time keys consumed)
POST /v1/media/uploads              -> { uploadUrl, mediaId }   (client encrypts before upload)
GET  /v1/media/{mediaId}            -> 302 to CDN signed URL
POST /v1/groups                     { name, members[] }
POST /v1/groups/{id}/members        { add[], remove[] }
POST /v1/reports                    { messageIds[], reason }   (reported messages decrypted by the reporter's device and shared)

Connection frame example (logical):

{ "type": "MSG", "clientId": "c-7f2a", "to": "u:91873", "conv": "d:91873:44102",
  "ts": 1795200000123, "body": "<ciphertext base64>", "kind": "text" }

12. Database schema and data model

Offline store (Cassandra/ScyllaDB CQL):

CREATE TABLE pending_messages (
  device_id   text,
  seq         timeuuid,
  sender_id   text,
  conv_id     text,
  client_id   text,
  payload     blob,          -- ciphertext
  PRIMARY KEY ((device_id), seq)
) WITH CLUSTERING ORDER BY (seq ASC)
  AND default_time_to_live = 2592000;   -- 30 days

Group membership (relational or wide-column): group_id, member_id, role, joined_at. Session registry: user_id → [(device_id, gateway_node, conn_id, last_seen)] in Redis-like memory store with TTL.

13. Database technology selection and reasoning

Data Store Reason
Pending messages Cassandra/ScyllaDB (or HBase-like) Massive write/delete throughput, partition per device, multi-DC replication
Session registry In-memory sharded store (Redis Cluster / custom) Microsecond lookups; data is reconstructible
Groups, accounts Sharded MySQL/PostgreSQL or distributed SQL Relational, moderate volume, consistency for membership
Key directory Strongly consistent replicated store Wrong keys break security
Media Object storage + CDN Cheap, durable, offloads bandwidth

Note on TTL deletes: wide-column stores create tombstones on delete; design compaction strategy (e.g. time-window compaction) and avoid reading over many tombstones.

14. Authentication and authorization

  • Phone number verification via SMS/voice OTP, then device key pair generated on device; device registered with public key.
  • Connection authentication with a device-signed challenge (no passwords).
  • Two-step verification PIN optional; registration lock to prevent SIM-swap hijacking.
  • Authorization: users can only send to non-blocked users; group admins control membership; business API uses OAuth tokens per business.

15. Security architecture

  • End-to-end encryption (Signal Protocol: X3DH/PQXDH key agreement, Double Ratchet for forward secrecy; Sender Keys for groups). The server stores only ciphertext and minimal routing metadata.
  • TLS (or Noise protocol) on transport in addition to E2EE.
  • Abuse prevention without reading content: rate limits, metadata signals (account age, send velocity, forwarding counts), user reports.
  • Media encrypted client-side with a random key sent inside the E2EE message.
  • Minimal retention: delete delivered messages from servers; short metadata retention.

[!VERSION] Signal introduced PQXDH (post-quantum key agreement) in 2023; messaging providers are progressively adopting post-quantum protections. Check current protocol versions.

16. Caching strategy

  • Session registry is effectively a cache rebuilt from reconnects.
  • Profile names/avatars cached on device and CDN.
  • Group membership cached in fan-out workers with invalidation on change events.

17. Messaging and event-driven architecture

  • Internally, routers publish to per-gateway queues or use direct RPC to the gateway node; offline path writes to the store.
  • Large group messages: sender encrypts once with Sender Keys; server fans out the same ciphertext to N device queues asynchronously.
  • Push notifications via queue to workers with retry and provider-specific rate limiting.

18. Cloud deployment architecture

  • Multiple regions; users connect to the nearest region via GeoDNS/anycast.
  • Cross-region routing when sender and recipient are in different regions: router forwards to the recipient's home region.
  • Offline store replicated across DCs in the user's home region; gateway nodes stateless apart from live connections.
  • Often bare-metal or large VMs for gateways due to connection density and kernel tuning.

19. CI/CD and DevOps

  • Hot code upgrades (Erlang) or connection-draining rolling deploys: tell clients to reconnect gradually with jitter to avoid reconnect storms.
  • Client protocol versioning with long backward compatibility (old app versions stay in the field for years).
  • Staged rollouts per region and per percentage of connections.

20. Monitoring and observability

  • Per gateway: connections, connect rate, memory per connection, message rate.
  • End-to-end delivery latency measured with synthetic clients across regions.
  • Offline queue depth and age; push success rates by provider.
  • Reconnect storm detection after network events.

21. Scalability and performance optimization

  • Kernel and runtime tuning for millions of sockets per node (file descriptors, buffer sizes, keepalive intervals).
  • Batch writes to the offline store; delete in batches on ack.
  • Group fan-out in workers with per-group partitioning; very large groups (communities/channels) use pull-based delivery.
  • Compression and compact encoding to save bandwidth.

22. High availability and disaster recovery

  • Gateway failure: clients reconnect to another node (with jittered backoff); pending messages remain in the store.
  • Router stateless; registry rebuilt on reconnect.
  • Region failure: clients reconnect to another region; offline store replicated across DCs; messages not yet acked by the server are resent from device outboxes.

23. Cost optimization

  • Connection density per node is the main lever (Erlang-style lightweight processes).
  • Delete messages after delivery instead of storing history (device is source of truth).
  • Media expires from servers after download/retention window; CDN offload.
  • Efficient binary protocol reduces bandwidth cost.

24. Architecture trade-offs and alternatives

Decision Chosen Alternative Trade-off
History Device-held, server deletes after delivery Server-stored history (Slack/Telegram cloud chats) Privacy vs multi-device convenience and search
Transport Custom persistent protocol MQTT, WebSocket Efficiency vs standard tooling
Groups E2EE Sender Keys Pairwise encryption per member Scale vs stronger per-recipient properties
Ordering Per conversation sequence Global ordering Global order is unnecessary and costly

25. Technology stack recommendations

Erlang/Elixir (or Go/Rust) gateways; Cassandra/ScyllaDB offline store; Redis-like registry; MySQL/PostgreSQL shards or distributed SQL for accounts and groups; object storage + CDN; Kafka for analytics and abuse signals; Signal Protocol libraries (libsignal); APNs/FCM; Prometheus/Grafana.

26. Common architecture mistakes

  1. Using HTTP polling for real-time messaging at scale.
  2. Storing undelivered messages in the gateway's memory (lost on crash).
  3. No client message IDs, causing duplicates on retries.
  4. Fan-out on the sender's request path for large groups.
  5. Synchronized reconnect storms after deployments or outages.
  6. Logging message content or keys in server logs.

27. Senior architect interview questions and answers

How do you guarantee ordering in a conversation?

Assign a monotonically increasing sequence per conversation at a single owner (the conversation's partition or the router responsible for it), store messages with that sequence, and have clients display by sequence and buffer gaps briefly. Different conversations do not need a global order.

How does a message reach a user who is offline for three days?

The server persists it in the device's pending queue (TTL e.g. 30 days) and sends a push notification. When the device reconnects, it fetches pending messages in order, acks them, and the server deletes them and sends delivered receipts.

How do you handle 200 M persistent connections?

Many gateway nodes, each holding hundreds of thousands to ~1 M connections with lightweight per-connection processes, efficient keepalive intervals, a session registry for routing, and connection-aware rolling deployments.

28. Enterprise architect interview questions and answers

How would E2EE affect compliance requirements for a business messaging product?

Content is not available to the platform, so retention/eDiscovery must happen at the business endpoint (the business's own systems receive and store messages). Architecture must document data flows, metadata retention and lawful access obligations per jurisdiction.

Build vs buy for an internal enterprise chat?

Buy (Teams, Slack) unless messaging is the product. For customer-facing chat inside an app, use a CPaaS or chat SDK unless scale/cost or privacy differentiation justifies building.

29. CTO-level business and technology questions

How do you keep cost per user low at 500 M DAU?

Extreme connection density, deleting delivered data, CDN offload for media, compact protocols, and a small, senior engineering team with deep runtime expertise. Unit cost per DAU per year should be tracked.

What are the main regulatory risks?

Lawful interception demands vs E2EE, data localization laws, content moderation obligations, and child safety regulations. These need policy and architecture positions decided at the executive level.

30. Real-world implementation roadmap

Phase Duration Outcomes
1. MVP 0–4 months 1:1 messaging, offline queue, push, single region
2. Groups and media 3–7 months Groups, encrypted media, receipts
3. E2EE everywhere 6–10 months Signal Protocol, key verification, multi-device
4. Multi-region scale 9–15 months Regional gateways, cross-region routing, load tests to 3× peak
5. Business platform 12–20 months Business API, templates, billing, abuse tooling