Everything in this module, assembled once, for a service you actually run. The pipeline is half the deliverable; the drills are the other half, because a gate that has never been tested is a gate you are assuming.
Topic 1: The Target
Pick a real service with a container image, a test suite, and somewhere to deploy. Any platform: Jenkins, Cloud Build plus Cloud Deploy, or a mix.
The requirements, stated as constraints rather than as a stage list:
1. The artifact deployed to production is the artifact that was tested —
same digest, provable.
2. No credential that can reach production is available to a pull-request build.
3. Every gate either blocks or does not exist. Nothing warns and proceeds.
4. Production requires an approval, and who approved it is recorded.
5. Rollback needs no rebuild, and you have timed it.
6. The pipeline definition lives in the repository it builds.
7. A stage failure names the cause without anyone reading application logs.
Constraint 1 is the one most existing pipelines fail, and constraint 3 is the one most “secure” pipelines fail.
Topic 2: Phase 1 — Build and Verify
pipeline {
agent none
options {
timeout(time: 40, unit: 'MINUTES')
buildDiscarder(logRotator(numToKeepStr: '50'))
disableConcurrentBuilds(abortPrevious: true)
timestamps()
}
environment {
REGISTRY = 'europe-west1-docker.pkg.dev/acme/apps'
IMAGE = "${REGISTRY}/checkout"
}
stages {
stage('Build') {
agent { kubernetes { yamlFile 'ci/agent-pod.yaml'; defaultContainer 'maven' } }
steps {
sh 'set -euo pipefail; mvn -B -DskipTests package'
stash name: 'app', includes: 'target/*.jar'
}
}
stage('Verify') {
parallel {
stage('Unit') { agent { … }; steps { sh 'mvn -B test' }
post { always { junit 'target/surefire-reports/*.xml' } } }
stage('Integration') { agent { … }; steps { sh 'mvn -B verify -Pit' } }
stage('Lint') { agent { … }; steps { sh 'mvn -B spotless:check' } }
}
}
}
}
Decisions to write down here, because a reviewer will ask:
- Why
agent noneat the top, and which stages hold an executor. - What is stashed versus published, and why.
- Which stages run on a pull request and which do not.
- Where the cache lives and what happens when it is poisoned.
Topic 3: Phase 2 — Image, Scan, Sign
stage('Image') {
agent { kubernetes { yamlFile 'ci/agent-pod.yaml' } }
steps {
unstash 'app'
container('kaniko') {
sh '''
set -euo pipefail
/kaniko/executor --context=. --dockerfile=Dockerfile \
--destination=${IMAGE}:${GIT_COMMIT} --cache=true \
--digest-file=/workspace/digest
'''
}
script { env.DIGEST = readFile('/workspace/digest').trim() }
echo "built ${env.IMAGE}@${env.DIGEST}"
}
}
stage('SBOM and scan') {
steps {
sh '''
set -euo pipefail
syft "${IMAGE}@${DIGEST}" -o spdx-json > sbom.json
grype sbom:sbom.json --fail-on high --only-fixed -o table
'''
archiveArtifacts 'sbom.json'
}
}
stage('Sign and attest') {
steps {
sh '''
set -euo pipefail
cosign sign --yes "${IMAGE}@${DIGEST}"
cosign attest --yes --predicate sbom.json --type spdxjson "${IMAGE}@${DIGEST}"
'''
}
}
Three properties to prove rather than assume: no Docker socket is mounted anywhere (ls -la /var/run/ in the build pod), the digest — not the tag — is what flows onward, and the scan genuinely fails the build when you plant a vulnerable dependency.
Topic 4: Phase 3 — Deploy, Gate, Promote
stage('Deploy staging') {
when { branch 'main'; not { changeRequest() } }
steps {
sh '''
set -euo pipefail
gcloud deploy releases create checkout-${GIT_COMMIT:0:7} \
--delivery-pipeline=checkout --region=europe-west1 \
--images=checkout=${IMAGE}@${DIGEST}
'''
}
}
stage('Smoke staging') {
when { branch 'main' }
steps { sh './ci/smoke.sh https://checkout.staging.acme.example' }
}
stage('Promote to production') {
when { branch 'main' }
steps {
// Cloud Deploy owns the approval — it is audited and outlives the build
sh '''
gcloud deploy releases promote --release=checkout-${GIT_COMMIT:0:7} \
--delivery-pipeline=checkout --region=europe-west1
'''
}
}
Put the approval in the deployment system, not in the build. A Jenkins input holds a build for hours, is lost if the controller restarts, and its record lives in a build log that rotates. A Cloud Deploy approval is an object with an IAM identity and an audit-log entry — the answer to “who authorised this” six months later.
If your platform has no such object, then a Jenkins input with submitter, submitterParameter and a timeout is the fallback, and the approver must be captured.
Topic 5: Phase 4 — The Nine Drills
Each drill: establish the state, run it, verify with a command, record the time.
1. A failing unit test blocks the merge. Break one test, push to a branch, open a PR. Expect: the PR check is red and the merge button is blocked by branch protection. Verify: the forge shows the required check failing.
2. A high CVE blocks the build. Add a dependency with a known fixable high-severity CVE. Expect: the scan stage fails. Then add it to your allow-list with an owner and an expiry, and confirm it passes — both halves matter.
3. An unsigned image is refused. Push an image built outside the pipeline and try to deploy it. Expect: admission control rejects it. Verify: the exact rejection message from the cluster.
4. The staging smoke test fails → no promotion. Make /healthz return 500 in staging. Expect: the promotion never happens. Record: how long until the pipeline noticed.
5. The approval is denied. Reject the production rollout. Expect: nothing ships and the release stays available for later. Verify: the audit log entry naming who denied it.
6. A canary error rate triggers rollback. Deploy a version returning 500 for a fraction of requests. Expect: the canary analysis fails and traffic returns to stable automatically. Record: time from first bad response to full rollback. If it did not catch it, write down the query that would have.
7. The same commit rebuilt produces the same digest. Re-run the build on the same commit. Expect: an identical digest, or a written explanation of exactly which input was not pinned.
8. No secret appears in any log. Grep the full console output of every stage for the last four characters of each credential. Expect: nothing. If something appears, fix the quoting and rotate the credential.
9. Rollback to the previous release, timed. From the decision to traffic on the old version. Record: the number. This is the most important figure in the whole project.
Topic 6: What to Produce, and How It Is Judged
Four artifacts:
- The pipeline definition in Git — Jenkinsfile,
cloudbuild.yaml, delivery pipeline and target YAML, plus theci/scripts every stage calls. - A drill log — nine drills, each with the command, the observed result, the verification command and the elapsed time.
- Four DORA numbers, measured over a week of real use: deployment frequency, lead time, change failure rate, time to restore.
- A fail-open/fail-closed note — for every gate in the pipeline, does it block or warn when it errors? A scanner that cannot reach its database and exits zero is a gate that fails open, and that is a decision, not an accident.
The acceptance test: hand someone a commit SHA. From the pipeline alone they can tell you what was built, what is inside it (the SBOM), who approved it, where it is running, and how to undo it. If any of those five requires asking a person, that is the gap to close.
The questions you should be able to answer without notes:
- Which of your gates fail open, and was that deliberate?
- What can a pull request from a fork reach?
- How long is your rollback, measured this month?
- If the CI platform were unavailable for a day, could you deploy a hotfix? How?
- Which stage is your longest, and what have you tried?
- What is deployed in production right now, and which commit is it?
Common mistake: building the pipeline, watching it deploy successfully, and calling it finished. Drills 2, 3, 5, 6 and 8 all fail on pipelines that build and deploy perfectly — the scan that warns, the admission policy in audit mode, the approval nobody records, the canary with no analysis, and the secret in a set -x trace. The green build proves the happy path; the drills prove the gates.