SkillPilotAI

CI/CD and Deployment Strategies

An automatic pipeline that builds, tests and deploys from code commit to production, plus ways to release a new version without downtime.

47.1 Why is it used?

  • Manual deploys are error-prone and slow; with CI/CD you can deploy safely 10-100 times a day.
  • CI (Continuous Integration): automatic build + unit tests on every commit, so bugs are caught early.
  • CD (Continuous Delivery/Deployment): a tested build moves automatically to staging/production.
  • With the right deployment strategy (canary, blue-green) a new bug hits only a few users and the whole site does not go down.
  • With a fast rollback, the damage of a bad release is contained within minutes.

47.2 How it works

  • Pipeline stages: Checkout -> Build -> Unit test -> Security scan -> Docker image push -> Deploy staging -> Integration test -> Deploy prod.
  • Rolling update: replace old instances with the new version batch by batch; no extra infra needed, but rollback is slow.
  • Blue-Green: two identical environments; test the new version on Green, then switch the load balancer/DNS so all traffic goes to Green; rollback = go back to Blue.
  • Canary: send 1-5% of traffic to the new version first, and if metrics (error rate, latency) look fine, raise it to 25%, 50%, 100%.
  • Feature flags: the code is deployed but the feature flag stays OFF; later turn it ON/OFF without a deploy, or ON for only some users.
  • GitOps (ArgoCD): the Git repo is the source of truth; ArgoCD keeps the cluster state matching the YAML in Git.
  • Rollback: redeploy the old image tag (kubectl rollout undo) or turn the feature flag OFF; DB migrations must stay backward compatible.
CI/CD pipeline: checkout service, commit to prod
  1. 1.PR: GitHub Actions test, lint, SAST
  2. 2.Build: Docker image
  3. 3.ECR: tag v2.4.0
  4. 4.Git: update image tag in Helm values
  5. 5.ArgoCD: staging sync + integration test
  6. 6.Prod: Argo Rollouts canary

CI = build + test on every commit. CD = a tested build goes automatically to staging/prod. GitOps: Git = source of truth.

Canary release: v2.3.9 stable to v2.4.0 new
  1. 1.Ingress / LB traffic split: 95% v2.3.9 stable, 5% v2.4.0 canary
  2. 2.Prometheus check for 10 min: error rate < 0.1%? p99 latency OK?
  3. 3.Yes: promote 25% -> 50% -> 100%
  4. 4.No: auto abort, 100% traffic back to v2.3.9

Smallest blast radius: a bug reaches only 1-5% of users.

Rolling vs Blue-Green vs Feature flag
  1. 1.Rolling: step 1 v1 v1 v1 v1 -> step 2 v2 v1 v1 v1 -> step 3 v2 v2 v2 v1; no extra infra, K8s default, slow rollback, mixed versions run together for a while
  2. 2.Blue-Green: the Load Balancer switches from Blue v1 (old) to Green v2 (tested); rollback = back to Blue in seconds, 2x infra, keep DB migrations backward compatible
  3. 3.Feature flag: v2.4.0 deployed with the 'one-click pay' code inside, Flag OFF (for nobody) -> Flag ON (at 12 midnight); deploy and release are separate, rollback = flag OFF instantly (LaunchDarkly / Unleash)

47.3 Tools

ToolTypeHow it is used
JenkinsOpen-source CI/CD serverSelf-hosted, pipeline via a Jenkinsfile; 1800+ plugins, very common in older enterprise setups.
GitHub ActionsHosted CI/CDYAML in .github/workflows in the repo; build-test-deploy on push/PR, ready-made marketplace actions available.
GitLab CIHosted / self-hosted CI/CDPipeline via .gitlab-ci.yml; GitLab repo, registry and security scans all in one place.
ArgoCDGitOps CD for KubernetesSyncs the K8s cluster as soon as YAML changes in Git; Argo Rollouts adds canary/blue-green.
AWS CodePipeline / CodeDeployAWS managed CI/CDRolling and blue-green deploys on EC2, ECS, Lambda, with automatic rollback on an alarm.
LaunchDarkly / UnleashFeature flag serviceTurn a feature ON/OFF by percentage, region or user group, without a redeploy.

47.4 Comparison: Rolling vs Blue-Green vs Canary vs Feature Flags

PointRollingBlue-GreenCanaryFeature Flags
How it worksReplace instances in batchesA whole new environment, switched in one goA little traffic first, then raise slowlyAn if-condition in code, ON/OFF at runtime
Extra infraNo2x infra (temporary)A little extraNo
Rollback speedSlow, another rolling updateSeconds, traffic goes back to BlueFast, canary traffic to 0%Instant, flag OFF
Risk / blast radiusMedium, mixed versions run togetherAll users at the moment of the switchLowest, only 1-5% of usersVery low, only chosen users
ComplexityLow, K8s defaultMedium, DB sync is trickyHigher, needs metrics + traffic splitMedium, flag cleanup is a must
Best forNormal small releasesBig releases that need instant rollbackRisky changes, high traffic sitesBusiness feature launches, A/B tests

47.5 When to choose which

  • GitHub Actions/GitLab CI: easiest for CI if your code is hosted there; Jenkins when you are on-prem and need heavy customisation.
  • ArgoCD: on Kubernetes, GitOps is the best CD; what is deployed is visible in Git.
  • Canary: when traffic is huge and one bug can cost crores (payment, checkout).
  • Blue-Green: when you need instant rollback and can afford 2x infra temporarily.
  • Feature flags: when deploy and release must be separate, for example turning a sale feature ON at 12 midnight.

47.6 Real-world example

A canary release of the checkout service (an illustrative scenario, Amazon-like).

  1. A developer opens a PR; GitHub Actions runs unit tests, lint and a SAST scan.
  2. On merge a Docker image is built and pushed to ECR with the tag v2.4.0.
  3. The pipeline updates the image tag in the Helm values in the Git repo; ArgoCD sees the change and syncs staging.
  4. After automated integration tests pass on staging, Argo Rollouts starts the canary in prod: 5% of traffic on v2.4.0.
  5. For 10 minutes Prometheus checks error rate and p99 latency; if the error rate is under 0.1% it goes 25%, 50%, 100%.
  6. If 5xx errors rise even at the 5% stage, Argo Rollouts aborts automatically and sends 100% of traffic back to v2.3.9.
  7. The new 'one-click pay' feature is in the code but the flag is OFF; on sale day at 12 midnight the product team turns the flag ON.

47.7 Interview tips

  • Explain the difference between Continuous Delivery (prod deploy with manual approval) and Continuous Deployment (fully automatic).
  • Common mistake: not keeping DB schema changes backward compatible; after a rollback the old code breaks on the new schema (use the expand-contract pattern).
  • Running a canary without an automatic metrics check is pointless, since nobody is watching.
  • Never deleting old feature flags turns the code into a jungle of flags. Number to quote: canary 5% -> 25% -> 50% -> 100%, promote while error rate < 0.1%.