CI/CD Pipeline Design: The Stages Every Deployment Needs

Fail fast, fail cheap — if a stage takes more than a few minutes, split it so the fast part runs first, or run independent jobs in parallel inside the stage.

One pipeline definition — keep it in the repository next to the code, so a change to the build and the change that needs it land in the same review.

Tag by commit, not by branch — a tag like :main or :latest moves; :3f9c2ab does not. Every later stage refers to the commit tag, and you can always trace a running container back to its source.

Configuration is not the artifact — environment differences belong in runtime config — environment variables, mounted files, secrets — never in build arguments that bake staging URLs into the image.

Pin build tooling — use docker:27 rather than docker:latest and pin base images in the Dockerfile. Unpinned tooling is the most common reason yesterday's green pipeline is red today with no code change. Our Dockerfile generator produces pinned multi-stage builds.

Pushing before tests is fine — the commit-tagged image is just a candidate. What matters is that only images that passed later stages get promoted or deployed.

--exit-code 1 — without it Trivy reports findings and exits 0, so the job always passes. --ignore-unfixed keeps the gate to vulnerabilities that actually have a patched version available.

Keep test reports on failure — artifacts: when: always uploads the JUnit file even when tests fail, which is exactly when you need it.

Flaky tests are pipeline bugs — a test that fails one run in twenty trains people to click retry. Quarantine it and fix it; do not add automatic retries to the test stage.

Scan the repo too — run a secret scanner such as gitleaks detect --source . in the lint stage. A leaked key in history needs rotation, not just a revert.

Restrict who can push — only the pipeline should be able to write to the production repository or tag pattern. If engineers can push images from laptops, the build-once guarantee is gone.

Retention — keep every image that was ever deployed to production for at least as long as you might need to roll back to it. Clean up untested branch images aggressively.

resource_group / concurrency — two pipelines deploying to production at the same time is how an older commit ends up live after a newer one. Serialize per environment.

rollout status is the real deploy step — kubectl set image returns as soon as the API accepts the change. rollout status blocks until new pods are ready or the timeout expires, and returns non-zero on failure. Readiness probes in your manifests decide what "ready" means — see the Kubernetes generator.

Smoke tests, not test suites — after deploy, hit a handful of critical endpoints: health, login, one read, one write. The goal is detecting a broken deploy in under a minute, not re-running the test stage.

curl -f matters — without -f, curl exits 0 on HTTP 500. --retry-all-errors (curl 7.71+) also retries on HTTP errors and refused connections while pods are still starting.

Expand, then contract — database migrations must work with both the old and the new version of the code. Add the new column first, deploy code that uses it, and drop the old column in a later release. Otherwise rollback breaks on the schema.

Mark deploys in your metrics — send a deploy event with the commit SHA to your monitoring system at the end of the job. When error rates jump, the first question is always what changed.

Practise the rollback — run it in staging on a schedule. A rollback path that has never been exercised fails the first time you need it.

Pipeline scripts are code — deploy logic in shell deserves the same review as application code. Our guide to reviewing shell scripts covers the quoting and error-handling bugs that bite in CI, and Docker for Linux admins covers running the resulting images.