Large File Upload and Download
Uploading/downloading GB-sized files (video, backups, documents) in pieces, in a resumable way, straight to object storage, without bringing the app server in the middle.
60.1 Why is it used?
- If you upload a 2 GB video in a single HTTP request and the network breaks at 90%, the user has to start again from 0.
- If the file goes through the app server, the server's bandwidth, memory and threads get blocked. A few uploads are enough to choke the server.
- Object storage (S3/Blob/GCS) handles bandwidth and scaling itself. Sending the client straight there is cheap and fast.
- Security: only an authorised user should upload/download, only for a limited time, and a malicious file (virus) must not get into the system.
- On the download side too you need resume, parallel ranges and fast delivery through a CDN.
60.2 How it works
- 1.Creator app -> Backend API: sends file name, size, type (auth + quota check, metadata DB)
- 2.Backend -> S3: CreateMultipartUpload, gets an uploadId
- 3.Backend -> App: 200 pre-signed URLs (video.mp4 = 200 parts x 10 MB, expiry 15 min)
- 4.App -> S3: PUT the parts directly, 4-6 in parallel (bucket: uploads/)
- 5.App -> Backend: done + ETags
- 6.Backend -> S3: CompleteMultipartUpload, a single object is created
If Wi-Fi drops and P3 fails, only P3 is retried; ListParts tells which parts are done; every part's checksum is verified
- 1.S3 Event (ObjectCreated)
- 2.SQS queue
- 3.Lambda scan: ClamAV + magic bytes (status: scanning)
- 4.Clean -> Transcoding 360p -> 720p -> 1080p
- 5.Infected -> Quarantine bucket, error to the user
- 6.Metadata DB status: uploaded -> processing -> ready
Mistake: streaming the file through the app server or loading it into RAM; a few uploads and the server goes OOM
- 1.Viewer player -> GET + Range request
- 2.CloudFront CDN (signed URL / cookie), returns a 206 chunk
- 3.On a cache miss it fetches from the S3 origin
- 4.Lifecycle rule: incomplete multipart uploads auto-delete in 7 days (hidden cost)
- Pre-signed URL: the backend checks auth and hands out a time-limited signed S3 URL (like 15 min); the client PUTs directly to S3.
- Multipart upload: split the file into parts (in S3 a part is min 5 MB, max 10,000 parts, max object 5 TB); upload parts in parallel and finish with CompleteMultipartUpload.
- Resume: only the part that failed is retried; the client knows which parts are done (ListParts) and skips them.
- tus protocol: an open HTTP-based resumable upload standard; the server reports the offset (HEAD request) and the client PATCHes from there.
- Integrity: verify each part's checksum (MD5/SHA-256/CRC) so corrupt data is not stored.
- After upload: a storage event (S3 Event -> SQS/Lambda) -> virus scan (ClamAV) -> if clean, to the 'final' bucket; if infected, to quarantine.
- Processing pipeline: thumbnail, video transcoding, save to the metadata DB; status 'uploaded -> scanning -> ready'.
- Download: a pre-signed GET URL or CDN signed URL; resume and parallel chunk download with HTTP Range requests.
60.3 Tools you can use
| Tool | Type | How it is used |
|---|---|---|
| Amazon S3 | Object storage | Pre-signed PUT/GET URLs, the Multipart Upload API, Transfer Acceleration, S3 Event Notifications for post-processing. |
| Azure Blob Storage (SAS) | Object storage | A SAS token URL with limited permissions and expiry; in block blobs, Put Block + Put Block List = chunked upload. |
| Google Cloud Storage | Object storage | Signed URLs and resumable upload sessions; parallel composite uploads for big files. |
| tus (tusd server, Uppy client) | Open resumable upload protocol | tusd server + Uppy/tus-js-client; when the network breaks it resumes automatically from the offset. |
| ClamAV / cloud malware scanning | Virus scanning | A ClamAV scan in a Lambda/Function after upload; or Microsoft Defender for Storage / GuardDuty Malware Protection for S3. |
| CloudFront / Azure Front Door / Cloud CDN | CDN | Fast downloads with signed URLs/cookies, Range request support, less origin load. |
| MinIO | S3-compatible self-hosted storage | The S3 API on-prem, with pre-signed URLs and multipart working the same way. |
60.4 Comparison: AWS S3 vs Azure Blob (SAS) vs Google Cloud Storage vs tus
| Point | AWS S3 | Azure Blob (SAS) | Google Cloud Storage | tus protocol |
|---|---|---|---|---|
| What it is | Managed object storage | Managed object storage | Managed object storage | An open protocol, not storage |
| Direct client upload | Pre-signed URL | SAS token URL | Signed URL | Through a tusd server |
| Chunking | Multipart (5 MB-5 GB parts, 10,000 max) | Blocks + Put Block List (50,000 blocks max) | Resumable upload / composite objects | PATCH chunks with offset |
| Resume | By tracking parts, ListParts | Through uncommitted blocks | Through the resumable session URI | Built-in core feature |
| Max object size | 5 TB | ~190 TB (block blob) | 5 TiB | Depends on the backend storage |
| Speed boost | Transfer Acceleration, parallel parts | Parallel blocks | Parallel composite | Depends on the client library |
| Best for | AWS stack, most popular | Azure/Microsoft stack | GCP stack, media/ML data | Cloud-neutral resumable UX (Vimeo, browser uploads) |
60.5 When to choose which
- S3/Blob/GCS pre-signed URL: the default choice. Bypasses the app server, direct upload, whichever cloud you are on.
- Multipart/chunked: always for files over 100 MB or on unreliable mobile networks.
- tus: when you want cloud-neutral resumable upload, large files from the browser, and control of your own server.
- MinIO: when on-prem/data-locality requirements need an S3-like API.
- For download always use CDN + signed URL + Range support; never a public URL for a private file.
60.6 Real-world example
Scenario: uploading a 2 GB video on a YouTube-like platform.
- The creator presses 'Upload': the app sends the file name, size and type to the backend; the backend does an auth + quota check.
- The backend calls CreateMultipartUpload and hands out pre-signed URLs for 200 parts (10 MB each), with a 15 minute expiry.
- The app uploads 4-6 parts in parallel and notes each part's ETag. If Wi-Fi drops, only the remaining parts are re-uploaded.
- After all parts, the app tells the backend; the backend calls CompleteMultipartUpload. A single object is created in S3.
- S3 event -> queue -> virus scan + file type check (magic bytes only, do not trust the extension).
- The clean file goes into the transcoding pipeline (multiple resolutions), and the status in the DB goes 'processing' -> 'ready'.
- Viewers get the video from the CDN; incomplete multipart uploads are auto-deleted after 7 days by a lifecycle rule.
60.7 Interview tips
- Streaming the file through the app server or loading the whole file into RAM: a few uploads and the server goes OOM.
- A very long pre-signed URL expiry, or not restricting content-type/size: it can be misused.
- Not cleaning up incomplete multipart uploads: hidden storage cost keeps growing in S3. Add a lifecycle rule.
- Interview tip: in 'Design Google Drive/Dropbox', explain the flow of pre-signed URL + chunking + dedup (chunk hash) + metadata DB + virus scan.