Background Jobs & Task Scheduling
Work that is not needed immediately in the user's request (email, report, resize) is done by a background worker, and work that must run at a fixed time is run by a scheduler.
27.1 Why is it used?
- If you do slow work like image resize, PDF invoice or video transcode inside the request, the API will take 10 seconds and time out.
- Jobs such as daily settlement, nightly report and subscription renewal must run automatically at a fixed time.
- Retry, backoff and failure tracking are managed in one place - if a job fails it tries again by itself.
- Web servers stay light; heavy work scales on separate worker machines.
27.2 How it works
- Job queue pattern: the app puts a job into a broker (Redis/RabbitMQ/SQS), and a worker process picks it up and runs it.
- Retry with exponential backoff: on failure retry after gaps of 1s, 2s, 4s, 8s..., and after max attempts the job goes to a dead/failed set.
- Delayed jobs: work like 'send the cart-abandonment reminder after 30 min' is scheduled with a future time.
- Cron expression: '0 2 * * *' means every day at 2 AM; the standard format for recurring jobs.
- Distributed lock / leader: if cron runs on 10 servers the job should not run 10 times, so use a lock (Redis/DB) or a single scheduler.
- Workflow / DAG: in Airflow, a dependency graph of task A -> B -> C, with retry and monitoring for each step.
- Idempotency: the result stays the same even if a job runs twice (e.g. check an 'already sent' flag).
- Priority queues: OTP/payment jobs go in a high-priority queue, reports in a low-priority one.
- 1.User login: needs an OTP
- 2.Auth API: enqueue, reply immediately
- 3.BullMQ (Redis): high-priority: OTP, low-priority: reports
- 4.OTP worker: < 1 sec, calls SMS provider A
- 5.Try 1 fails -> wait 1s -> Try 2 fails -> wait 2s -> Try 3 fails -> wait 4s
- 6.Fallback: OK via provider B
Exponential backoff: the gap doubles each time, and after max attempts the job goes to a failed set / DLQ
- 1.Extract transactions
- 2.Calculate settlement
- 3.Generate bank file
- 4.Upload to the bank
- 5.If settlement fails, only this step is retried + Slack alert to on-call
- 1.EventBridge Scheduler: cron, 1st of month
- 2.Lambda: puts 50 lakh job IDs into SQS
- 3.SQS: send only the ID, not the whole object
- 4.200 workers: PDF + email, auto scale down
- 5.DB flag check: statement_sent? yes -> skip
- 6.User inbox
Tools: Celery, Sidekiq, K8s CronJob. Mistake: the same cron on every app server -> the job runs N times; use a distributed lock (Redis/DB) or a single scheduler. Remember: retry + idempotency + DLQ - all three together
27.3 Tools you can use
| Tool | Type | How it is used |
|---|---|---|
| Celery | Python distributed task queue | In Django/Flask apps; broker is RabbitMQ or Redis, and Celery Beat handles periodic tasks. |
| Sidekiq | Ruby background jobs | Redis-based, multi-threaded workers in Rails apps - very fast and memory efficient. |
| BullMQ | Node.js job queue | Redis based; supports delayed, repeatable and priority jobs and rate limiting. |
| cron / Kubernetes CronJob | OS / K8s scheduler | Simple time-based scripts; a K8s CronJob runs a new pod on every schedule. |
| Apache Airflow | Workflow orchestrator | Write DAGs in Python; ETL/data pipelines, dependencies, backfill and UI monitoring. Managed: AWS MWAA, GCP Cloud Composer. |
| AWS EventBridge Scheduler + Lambda | Serverless scheduling | Triggers Lambda, SQS or Step Functions on a cron/rate or one-time schedule - no servers. |
27.4 Comparison: Celery vs BullMQ vs Airflow vs EventBridge Scheduler
| Point | Celery | BullMQ | Airflow | EventBridge Scheduler |
|---|---|---|---|---|
| Language | Python | Node.js / TypeScript | Python (DAG code) | Language independent (AWS targets) |
| Main job | Async tasks + periodic tasks | Async, delayed, repeatable jobs | Multi-step workflows / ETL pipelines | Time-based triggers |
| Backend | RabbitMQ / Redis / SQS | Redis | Metadata DB + executor (Celery/K8s) | Fully managed AWS |
| Latency | Milliseconds | Milliseconds | Seconds to minutes (batch oriented) | Minute-level schedule, one-time too |
| Dependencies | Chains/groups (basic) | Parent-child flows | Strong: full DAG, backfill | No, combine with Step Functions |
| Ops | Manage workers + broker | Redis + workers | Heavy (or MWAA/Composer) | Zero ops |
| Best for | Background work in a Python web app | Emails, webhooks in a Node app | Data engineering, nightly reports | Serverless cron, reminders |
27.5 When to choose which
- Celery: emails, image processing and async API calls in a Python/Django app.
- Sidekiq / BullMQ: fast Redis-based background jobs in a Ruby or Node app.
- Airflow: multi-step data pipelines that need dependencies, retries and backfill.
- EventBridge Scheduler + Lambda: serverless cron on AWS; on Kubernetes use a K8s CronJob.
- Plain cron: only a simple script on a single server - avoid it in a distributed setup.
27.6 Real-world example
A hypothetical scenario: daily statement and OTP in a fintech app
- On login the OTP job goes into the BullMQ 'high-priority' queue, and a worker calls the SMS provider within 1 second.
- If the SMS provider fails, BullMQ retries 3 times with exponential backoff, then uses a fallback provider.
- Every night at 1 AM an Airflow DAG runs: extract transactions -> calculate settlement -> generate bank file -> upload.
- When any step fails, Airflow retries only that step and sends a Slack alert to on-call.
- On the 1st of the month EventBridge Scheduler triggers a Lambda that puts statement jobs for 50 lakh users into SQS.
- 200 workers pick jobs from SQS, build PDFs and email them; when the queue empties the workers scale down.
- Each job checks the 'statement_sent' flag so a retry does not send a duplicate email.
27.7 Interview tips
- Common mistake: putting the same cron on every app server - the job runs N times; use a distributed lock or a single scheduler.
- Common mistake: putting the whole object in the job instead of the ID - data goes stale and the queue gets heavy; send only the ID and let the worker read fresh data. Not monitoring failed jobs is also wrong - they fail silently and the customer finds out first.
- Interview tip: say 'retry + idempotency + DLQ' all together.
- Worth quoting: exponential backoff 1s, 2s, 4s, 8s; cron '0 2 * * *' = every day at 2 AM; cron on 10 servers = the job runs 10 times.