Design a video streaming platform (YouTube-like) that lets users upload videos and stream them to millions of viewers across varying device types and network conditions, with fast global delivery.
For a platform with 500M daily active viewers, 500K video uploads/day:
Upload path (async, can tolerate minutes of latency):
[Client] → [Upload Service] → [Raw Video Object Storage]
│
▼
[Transcoding Pipeline] (message-queue-driven, see below)
│
▼
[Transcoded Video Object Storage] (multiple resolutions/bitrates)
│
▼
[CDN] (pre-warmed or pulled on first request)
Playback path (must be low-latency):
[Client Player] → [CDN edge node] → (cache miss) → [Origin Object Storage]
Raw upload → [Message Queue] → Transcoding Workers (horizontally scaled, CPU-heavy)
→ for each target resolution/bitrate (e.g. 240p, 480p, 720p, 1080p, 4K):
split into short segments (typically 2-10 seconds each)
encode each segment independently → upload to object storage
Transcoding is deliberately fully asynchronous and queue-driven (Message Queue System) — it's CPU-intensive and can take minutes for a long video, which is entirely acceptable since it's not on the critical path of any user-facing request; a message queue absorbing upload spikes and letting a horizontally-scaled worker pool process them at its own pace is the standard pattern.
The video is encoded into MULTIPLE bitrates and split into short segments; the player downloads a manifest file listing all available segments/bitrates and switches between bitrates SEGMENT BY SEGMENT based on currently observed network conditions — this is what lets playback continue smoothly (dropping to a lower bitrate) rather than buffering/stalling when a viewer's network briefly degrades, and it's the direct answer to the "adapt to network conditions" and "low startup latency" requirements (the player only needs to download one small segment to begin playing, not the whole file).
POST /api/v1/videos/upload (multipart, or a pre-signed direct-to-storage URL)
→ 202 { "videoId": "vid_123", "status": "PROCESSING" }
GET /api/v1/videos/{id}/manifest
→ 200 { "resolutions": ["240p","480p","720p","1080p"], "segmentUrls": {...} }
(an HLS .m3u8 or DASH .mpd manifest in practice)
GET /api/v1/videos/{id}/status
→ 200 { "status": "READY" | "PROCESSING" | "FAILED" }
Uploads use a pre-signed URL pattern (the client uploads directly to object storage using a short-lived signed URL issued by the Upload Service) rather than routing the full video file through the application server — avoiding the application tier becoming a bandwidth bottleneck for large file uploads.
videos
id, uploaderId, title, description, status, durationSeconds,
rawStorageKey, createdAt
video_renditions (one row per resolution/bitrate produced by transcoding)
id, videoId, resolution, bitrate, storageKeyPrefix, segmentCount
views (append-only, high volume — a natural fit for a wide-column or time-series store
rather than the primary relational-style video metadata store)
videoId, viewerId (or anonymized), timestamp, watchDurationSeconds
Splitting video_renditions from videos reflects the real data shape: one video has many renditions, produced asynchronously and at different times as transcoding completes — a viewer's manifest request needs to know which renditions are actually ready, not assume all of them exist immediately after upload.
The CDN is the primary scaling lever for playback (see Capacity Estimation's bandwidth numbers) — the origin object storage and application servers only need to handle CDN cache-miss traffic, a small fraction of total views for any popular video. The transcoding pipeline scales horizontally by adding more worker instances consuming from the queue — since transcoding jobs are independent per video (and per segment, within a video), this scales close to linearly with worker count. Video metadata (title, description, view counts) is read far more than written and benefits from aggressive caching, similar to Twitter/X's feed-caching approach.
Transcoding compute is the main internal bottleneck — it's CPU-intensive and scales with upload volume, not view volume, making it the part of the system most sensitive to upload spikes (a viral creator suddenly uploading many videos, or a platform-wide surge). CDN egress cost (not a technical bottleneck, but a very real economic one at this scale) is the other major constraint shaping design decisions like adaptive bitrate and on-demand transcoding for unpopular content.
A transcoding worker crashes mid-job: the message queue's redelivery (the job isn't acknowledged until fully complete) means another worker picks it up — this requires the transcoding job itself to be safely re-runnable (idempotent) if partially completed, similar in spirit to Payment — Idempotency Implementation's concerns, though the cost of a duplicate transcode is wasted compute, not incorrect money movement.
CDN edge node failure: transparent to the viewer — CDN providers route around a failed edge node automatically; the platform's own architecture doesn't need explicit handling for this, it's the CDN's responsibility.
A viewer's network degrades mid-playback: handled by adaptive bitrate switching itself (the intended, designed-for case, not really a "failure") — the player detects reduced throughput and requests lower-bitrate segments for subsequent playback without interrupting the current segment.