Log in
SkillPilotAI
Plans

Case Study 10: Google Docs Collaborative Editor

Many people edit the same document at once in real time; everyone sees the same final content, cursors are visible and older versions can be brought back.

C.1 Requirements

Functional requirements:

  • Create, open, edit and delete a document.
  • Multiple users edit in real time at once; changes show up for everyone in milliseconds.
  • Live cursors and presence (who is in the document, where they are typing).
  • Version history: view and restore older versions.
  • Sharing and permissions: owner, editor, commenter, viewer.
  • Comments and suggestions mode.
  • Offline editing and sync on reconnect.

Non-functional requirements:

  • Edit propagation latency < 200 ms within the same region.
  • Convergence: all clients must finally reach exactly the same document state.
  • Durability: an acknowledged edit must never be lost.
  • Support up to 100 concurrent editors on one document (assumption).
  • 99.9% availability; on a network drop the client keeps working in offline mode.
  • Crores of documents, total storage scaling to petabytes.

C.2 Estimation (back-of-the-envelope)

ItemNumberHow it is derived
Daily active users~5 croreAssumption
Concurrent open docs at peak~1 crore~20% of DAU at once, avg ~1 doc each.
WebSocket connections~1 croreOne server handles ~50,000 connections -> ~200 collab servers.
Edit ops per second~20 lakh/sec~10 lakh active typists x ~2 ops/sec (keystrokes batched).
Op log storage per day~10 TB20 lakh ops/sec x ~100 bytes x ~50,000 effective active sec/day.
Snapshot storage~50 KB per doc snapshotA snapshot every ~1000 ops or a few minutes, old ops are compacted.
Bandwidth per editor~2-5 KB/secOps + cursor updates, cursors throttled to ~10/sec.

C.3 API design

EndpointPurpose
POST /v1/docsCreate a new document
GET /v1/docs/{id}Load the latest snapshot + revision number
WS /v1/docs/{id}/collabWebSocket: send/receive ops, acks, cursor and presence events
GET /v1/docs/{id}/revisions?cursorVersion history list
POST /v1/docs/{id}/restoreRestore an older revision (a new revision is created)
PUT /v1/docs/{id}/permissionsUpdate the share settings

C.4 Data model

Table / StoreFieldsWhy
documents (Spanner/MySQL)doc_id PK, owner_id, title, latest_rev, snapshot_ref, created_at, updated_atA consistent relational store for metadata.
doc_operations (Bigtable/Cassandra, partition by doc_id)doc_id, revision (clustering), user_id, op_payload, timestampAn append-only op log, fast range reads in revision order.
snapshots (Object storage like GCS/S3)doc_id, revision, compressed_contentSo that loading a doc does not need replaying all the ops.
permissions (SQL + cache)doc_id, principal_id, roleA fast ACL check on every connection/edit.
presence (Redis, TTL)doc_id, user_id, cursor_pos, color, last_heartbeatEphemeral data; no need for durable storage.

C.5 High-level architecture

Google Docs: architecture and request flow
  1. 1.Clients: Web Editor (local state + rev) and Mobile App (offline queue); static JS from the CDN
  2. 2.Global Load Balancer -> API Gateway (REST: GET /docs/{id}) and WebSocket Gateway (sticky: consistent hash on doc_id)
  3. 3.Permission Service (owner / editor / viewer): ACL check on connect and on every op; viewer ops rejected
  4. 4.Doc Service (load = snapshot + later ops); Collab Server (OT engine, 1 leader per doc); Presence (cursor, throttled)
  5. 5.Stores: Metadata DB (Spanner / MySQL: documents, permissions), Snapshot Store (GCS / S3, ~50 KB / snapshot), Op Log Store (Bigtable / Cassandra, partition doc_id, rev order), Redis (presence TTL + routing)
  6. 6.Kafka: accepted op events -> Snapshot Compactor (new snapshot every ~1000 ops), Search Indexer (doc text index), Notification Worker (comment / share alerts)

Ephemeral vs durable: presence / cursor in Redis TTL, only broadcast; ops durably appended to the op log. Real-time edit path: ack + broadcast back over the WebSocket. Scale: 5 crore DAU, 1 crore WS connections, 20 lakh ops/sec, edit < 200 ms.

  • The web / mobile editor fetches static JS from the CDN; opening a doc via REST goes through the API Gateway, live edits through the WebSocket Gateway; both sit behind the Global LB.
  • The WebSocket Gateway sends all users of the same doc to a single Collab Server (the doc leader, OT engine) using a consistent hash(doc_id).
  • Doc Service load = latest snapshot (GCS / S3) + later ops (Op Log); the Permission Service checks the role on every connect / op.
  • An accepted op is durably appended to the Op Log, then acked + broadcast, then Kafka -> Snapshot Compactor, Search Indexer, Notification Worker.
  • Presence / cursor is ephemeral (Redis TTL), broadcast only; ops and snapshots go to durable stores.
ComponentWhat it does
WebSocket GatewayHandles long-lived connections; routes all users of the same doc to the same Collab Server using consistent hashing.
Collab ServerThe single owner/leader of every document. Transforms incoming ops against the server revision (OT), sequences them and broadcasts.
Doc ServiceManages doc create, load (snapshot + later ops), metadata and title.
Presence ServiceCursors and online users in Redis with a TTL; updates are throttled and only broadcast, not stored.
Op Log StoreEvery accepted op is durably appended with a revision number; this is the base of version history.
Snapshot CompactorPeriodically merges ops into a snapshot so loading is fast and the log stays small.
Permission ServiceChecks the role on connect and on every op; viewer ops are rejected.

C.6 Main flows

Opening a document:

  1. The user does GET /docs/{id}; the Permission Service checks the role.
  2. Doc Service fetches the latest snapshot from object storage and the ops after it from the op log.
  3. The client builds its local state and remembers the current revision number.
  4. The client connects over a WebSocket; the gateway picks the right Collab Server from the doc_id hash.
  5. The Presence Service broadcasts a join event and the other users see the new cursor.

Real-time edit (OT) flow:

  1. User A types; the op is applied locally at once (optimistic UI).
  2. The client sends the op: insert(pos=10, "a") at base revision 41.
  3. The Collab Server sees that the server revision is 43, so it transforms the op against the ops of rev 42 and 43.
  4. The transformed op is assigned revision 44 and durably appended to the op log.
  5. A gets the ack, and the transformed op is broadcast to the other clients.
  6. Each client transforms the incoming op against its own pending (unacked) ops and applies it.
  7. All clients converge on the same content.

Version restore flow:

  1. The user opens the history panel; revisions are shown grouped by time and author.
  2. Clicking a revision: the content at that time is built by replaying ops over the nearest snapshot.
  3. On restore, a new op/revision is created that replaces the current content.
  4. History is never deleted; a restore is also a new version.

C.7 Deep dives

Real-time edit OT: server rev 43, A's op is on base rev 41
  1. 1.User A client: types "a", local apply (optimistic UI); op insert(pos=10, "a") @41
  2. 2.Collab Server doc leader: the server is at rev 43, so the op is transformed against rev 42, 43; pos 10 -> 12, assign rev 44
  3. 3.Server op log (doc_id partition, revision order): rev 41 is where A has seen up to; rev 42 B: insert(3, "xy"), pos 3 < 10 so A's pos +2; rev 43 C: delete(20), pos 20 > 12 so no effect; rev 44 A: insert(12, "a")
  4. 4.Op Log: rev 44 durably appended, then ack -> A (pending op cleared, rev = 44)
  5. 5.Broadcast insert(12, "a") -> B, C: they transform it against their pending ops and apply it
Why is transform necessary? doc = "abc"
  1. 1.A: insert(0, "x"), local: "xabc"; B: delete(2) = "c", local: "ab"
  2. 2.Without OT: delete(2) on A removes "b" -> "xac"; insert(0, "x") on B applied as-is -> "xab"
  3. 3.They diverge: both have different content
  4. 4.OT: on A delete(2) -> delete(3) -> "xab"; on B insert(0, "x") as-is -> "xab"
  5. 5.They converge: everyone has the same content
Client state + offline reconnect
  1. 1.Client: confirmed rev = 44 (acked by the server) + pending unacked ops (applied locally, ack awaited)
  2. 2.An incoming op from another user: transform against pending ops -> apply -> increment rev
  3. 3.Offline: pending ops in IndexedDB
  4. 4.Reconnect: send the last acked rev; the server returns the missing ops; the client rebases + sends

OT vs CRDT: OT = central server transform (Google Docs), simple. CRDT (Yjs, Automerge) = unique char ID, offline / P2P.

Problem: conflicts from concurrent edits, OT vs CRDT. OT transforms ops with a central server; Google Docs uses an OT-based approach (public information). CRDT (Yjs, Automerge) gives every character a unique ID and converges without a central server, which is good for peer-to-peer/offline, but has more metadata overhead.

Problem: all users of one doc on one server. Consistent hashing on doc_id; the routing table lives in Redis/ZooKeeper. If a server fails the doc moves to a new server, which rebuilds the state from the op log, and clients reconnect and fetch the missing ops.

Problem: offline editing and reconnect. The client queues pending ops locally (IndexedDB). On reconnect it sends the last acked revision, the server returns the missing ops, and the client rebases/transforms and sends its own ops.

Problem: the load of cursor/presence traffic. Throttle cursor updates (for example 50-100 ms), do not write them to a durable store, only broadcast in memory. In large docs send only the cursors in the visible area.

Problem: slow loading because of a long history. A snapshot every N ops; load = latest snapshot + a few ops. Old ops go to cold storage and are fetched on demand for version history.

C.8 Trade-offs

DecisionWhat was chosen and why
OT vs CRDTFor an online editor with a central server, OT is simpler and memory-efficient; if you need offline-first/P2P, CRDT is better.
WebSocket vs long polling/SSEWebSocket: bidirectional, low latency. Keep long polling as a fallback for corporate proxies.
Single leader per doc vs multi-leaderSingle leader per doc: ordering is simple; the limit is that one doc's load stays on one server, which is practically fine.
Op log vs sirf snapshotsBoth: the op log gives fine-grained history and sync, snapshots give fast loading.

C.9 Interview tips

  • Show the convergence problem first with a simple example: two people type at the same position.
  • Explain both OT and CRDT and clearly say which you choose and why.
  • Draw the client state model of revision number + ack + pending ops.
  • Keep presence separate from durable data: the distinction between ephemeral and persistent.
  • Be sure to describe the flow of syncing missing ops during server failover and reconnect.