Log in
SkillPilotAI
Plans

Image & Video Processing / Streaming

Converting an uploaded image/video into different sizes and qualities so it plays smoothly on every device and network.

19.1 Why is it used?

  • A user uploads a 12 MP, 5 MB photo, but a mobile listing needs only a 300px thumbnail. Sending the original makes the page slow.
  • Every user's network is different (5G, 4G, slow WiFi); a single-quality video will buffer.
  • Different devices support different formats (H.264, H.265/HEVC, VP9, AV1; JPEG, WebP, AVIF), so conversion is necessary.
  • Processing is heavy CPU/GPU work, so it is done in async background workers to keep the upload API fast.
  • Good compression cuts both bandwidth and the CDN bill a lot.

19.2 How it works

  • Transcoding: converting a video from one codec/bitrate/resolution to another, for example renditions of 1080p, 720p, 480p and 240p from a 4K source.
  • Thumbnails / resizing: generating multiple image sizes (100px, 300px, 800px) and modern formats (WebP/AVIF).
  • Adaptive Bitrate Streaming (ABR): the video is cut into small 2-6 second segments, with separate segments for each quality.
  • HLS (Apple) uses a .m3u8 playlist and DASH (MPEG standard) uses a .mpd manifest; the player watches network speed and switches the quality of each segment.
  • Async pipeline: upload -> object storage -> queue/event -> transcoding workers -> output S3 -> CDN.
  • On-the-fly image transformation: write the size/format in the URL (for example w_300,f_auto), and the service resizes instantly and caches on the CDN.
  • DRM (Widevine, FairPlay, PlayReady): encrypting paid content to stop piracy, for OTT such as Netflix/Hotstar.
  • Live streaming: an encoder ingests over RTMP/SRT, the server creates HLS/DASH segments in real time, and the CDN delivers them to lakhs of viewers.
A. Live streaming pipeline (Hotstar IPL)
  1. 1.Stadium feed: live camera
  2. 2.Encoder: RTMP / SRT cloud ingest
  3. 3.Live transcoder: AWS MediaLive / FFmpeg, 1080p / 720p / 480p / 240p
  4. 4.HLS / DASH: 2-6 sec segments + .m3u8 / .mpd manifest
  5. 5.CDN (multi-CDN): segments cached, pushed
  6. 6.Viewers: 2.5 crore+ concurrent on mobile, TV, web players

After the match, the same content stays in S3 + CDN as VOD highlights

B. Adaptive Bitrate (ABR): the player picks each segment's quality itself
  1. 1.Player (Video.js / ExoPlayer): reads the .m3u8
  2. 2.seg 1-2: 1080p (5G / 4G)
  3. 3.seg 3: 720p (slow 4G)
  4. 4.seg 4: 240p (train, 2G) - network dropped
  5. 5.seg 5: 480p (3G)
  6. 6.seg 6: 1080p (WiFi)

If the network drops, the next segment is 240p: slightly lower quality, but no buffering

C. Async processing pipeline after upload (VOD + images)
  1. 1.Upload API: fast response
  2. 2.S3 raw: original 4K / 5 MB
  3. 3.Queue / Event: SQS / S3 event
  4. 4.Workers: FFmpeg / MediaConvert / sharp
  5. 5.S3 output: renditions, thumbs, WebP
  6. 6.CDN: serves users

Mistake: synchronous transcoding in the upload API -> request timeout; always use a queue + worker. Image sizes: 100 / 300 / 800 px, WebP / AVIF; on-the-fly via URL: w_300,f_auto

19.3 Tools you can use

ToolTypeHow it is used
FFmpegOpen-source media CLI/libraryTranscoding, extracting thumbnails, creating HLS segments; for example ffmpeg -i in.mp4 -vf scale=-2:720 out720.mp4. Runs on self-hosted workers.
AWS Elemental MediaConvertManaged video transcodingTake a video from S3, submit a job, and get HLS/DASH output with multiple renditions in S3; pay per minute.
CloudinarySaaS image/video platformUpload and then resize, crop, f_auto format and video ABR through URL parameters - CDN built in.
Azure Media Services / GCP Transcoder APIManaged cloud transcodingAzure/GCP equivalents of MediaConvert (Azure Media Services has been retired, partners are used now); GCP Transcoder API outputs HLS/DASH.
ImageMagick / libvips (sharp)Open-source image processingResizing and format conversion of images in a server/Lambda; sharp (libvips) is very fast and light on memory.
Video.js / Shaka Player / ExoPlayerABR video playersRead the HLS/DASH manifest on the client and switch quality according to bandwidth.

19.4 Comparison: FFmpeg vs AWS MediaConvert vs Cloudinary

PointFFmpeg (self-hosted)AWS MediaConvertCloudinary
TypeOpen-source tool, you run it on your own serversFully managed AWS serviceSaaS: storage + processing + CDN
ControlFull control, you can tune every flagPresets + good controlLess control, work is done via URL params
ScalingYou build workers, queue and autoscalingAutomatic scaleAutomatic scale
CostOnly compute cost, cheap at large scalePay per output minutePlan/credits based, expensive at large scale
Images supportBasic (frames/thumbnail)No, video onlyYes, best image support
Setup effortHigh, needs an ops teamMedium, needs AWS knowledgeVery low, start in minutes
Best forHuge scale like Netflix/YouTube, custom pipelineVOD/OTT library on AWSStartups, e-commerce images, quick launch

19.5 When to choose which

  • Cloudinary (or imgix): an e-commerce/startup needs images fast and has no media team.
  • AWS MediaConvert: an OTT/VOD platform on AWS that needs HLS/DASH + DRM without running its own cluster.
  • FFmpeg on Kubernetes/EC2 Spot: very large volume and you want full control over cost/quality.
  • sharp/libvips on Lambda: cheap and simple for creating basic thumbnails on an S3 upload event.
  • For live events (IPL, cricket) take a managed live service (AWS MediaLive) + multi-CDN.

19.6 Real-world example

Real world: Hotstar (JioHotstar) IPL live streaming

  1. The stadium's live feed goes to an encoder and reaches the cloud ingest over RTMP/SRT.
  2. The live transcoder creates HLS/DASH segments of multiple qualities (for example 240p to 1080p) every few seconds.
  3. Segments and the manifest are pushed to the CDN; Hotstar handled crores of concurrent viewers in IPL (public record 2.5 crore+ concurrency).
  4. The user's player downloads the .m3u8 manifest and chooses quality by watching the network.
  5. If the network drops on a train, the player picks the next segment at 240p - quality dips slightly instead of buffering.
  6. After the match, the same content is transcoded for VOD highlights and stays in object storage + CDN.
  7. Thumbnails and the player's scrub previews are made by extracting frames with tools such as FFmpeg.

19.7 Interview tips

  • Common mistake: doing synchronous transcoding inside the upload API - request timeout; always use a queue + background worker.
  • Common mistake: serving a single-quality MP4 - buffering on slow networks; use ABR (HLS/DASH). Sending the original 5 MB image everywhere is also wrong: generate thumbnails and WebP/AVIF.
  • Interview tip: if asked HLS vs DASH, say both are segment-based ABR; HLS is Apple's (native on iOS), DASH is the open MPEG standard.
  • Numbers worth quoting: 2-6 second segments, a ladder like 1080p/720p/480p/240p; Hotstar's public record of 2.5 crore+ concurrent viewers.