Credentials and Secrets in Jenkins

How a credential reaches a build step, the five ways log masking is defeated, and why OIDC federation removes the problem instead of managing it.

intermediate 20 min lesson hands-on task included

Jenkins holds credentials to everything: registries, clouds, repositories, production. It is therefore one of the highest-value targets in an engineering organisation, and the way pipelines use secrets decides how bad a compromise is.


Topic 1: How a Credential Reaches a Step

A CREDENTIAL IS BOUND FOR A BLOCK, MASKED IN LOGS — AND STILL LEAKABLE credential store scoped: global / folder / job binding environment {} or withCredentials the step sees an env var, for that block only THE TWO FORMS environment { AWS = credentials('aws-deploy') } — whole pipeline or stage withCredentials([usernamePassword(credentialsId: 'nexus', …)]) { … } — one block, preferred Narrower scope is better: the secret exists in the environment for fewer steps and fewer child processes. HOW MASKING IS DEFEATED · echo $SECRET | base64 → masking sees no match· set -x in a shell step → the command line prints· curl -v with the token in a URL· a test that dumps the environment· an agent shared with another job Masking is a courtesy, not a control. WHAT ACTUALLY REDUCES THE RISK · short-lived tokens over static keys· OIDC federation to the cloud — no stored key· credentials scoped to a folder, not global· ephemeral agents, destroyed after the build· no credential a pull-request build can reach A fork PR must never see a deploy credential.
Binding puts the secret in the environment for a block. The left panel is how that protection is routinely defeated; the right panel is what actually reduces the risk.

Credentials live in a store, scoped globally, to a folder, or to a job. A pipeline binds one into the environment for a block, and Jenkins attempts to mask it in the console output.

The two binding forms:

// environment block — the whole pipeline or a stage
environment {
    NPM_TOKEN = credentials('npm-publish-token')
}

// withCredentials — one block, and the form to prefer
steps {
    withCredentials([
        string(credentialsId: 'npm-publish-token', variable: 'NPM_TOKEN'),
        usernamePassword(credentialsId: 'nexus', usernameVariable: 'NEXUS_USER', passwordVariable: 'NEXUS_PASS'),
        file(credentialsId: 'gcp-sa-key', variable: 'GOOGLE_APPLICATION_CREDENTIALS'),
        sshUserPrivateKey(credentialsId: 'deploy-key', keyFileVariable: 'SSH_KEY')
    ]) {
        sh 'npm publish'
    }
}

withCredentials is narrower: the secret exists in the environment for those steps only, rather than for every step and every child process in the pipeline. On a long pipeline that difference is meaningful.

credentials() in an environment block has a quirk worth knowing: for a usernamePassword credential it creates three variables — NAME, NAME_USR and NAME_PSW. Code that expects one variable and finds a colon-joined pair is a common half-hour.

Credential types, and when each is right:

TypeUse
Secret textTokens, API keys
Username with passwordRegistries, repositories
Secret fileService-account JSON, kubeconfig, certificates
SSH username with private keyGit over SSH, deploys
CertificatemTLS client certificates

Topic 2: How Masking Is Defeated

Jenkins masks a credential by replacing exact matches of the value in the console output. That is a courtesy, not a control, and it fails in at least five ordinary ways:

// 1. Transformed — masking sees no exact match
sh 'echo $SECRET | base64'

// 2. Shell tracing prints the expanded command
sh 'set -x; curl -H "Authorization: Bearer $TOKEN" https://api.example'

// 3. Groovy interpolation puts it in the command line, visible to `ps`
sh "curl -H 'Authorization: Bearer ${TOKEN}' https://api.example"

// 4. A tool dumps its environment on error
sh './deploy.sh || env'

// 5. It is written to a file that is archived as an artifact
sh 'echo $TOKEN > .netrc'
archiveArtifacts '**/*'

Number 3 is the one to internalise: use single quotes so the shell expands the variable, not Groovy.

sh 'curl -H "Authorization: Bearer $TOKEN" https://api.example'   // ✓
sh "curl -H 'Authorization: Bearer ${TOKEN}' https://api.example" // ✗

Both work. Only the first keeps the value out of the rendered command line — which appears in the log, in ps output on a shared agent, and in the Jenkins pipeline step view.

A secret that has been printed is compromised. The response is rotation, not deleting the build log — the log has already been read by anyone with access, indexed by whatever ships your logs, and possibly copied into a support ticket.


Topic 3: Scoping, and the Pull Request Rule

Scope credentials to folders, not globally. A global credential is usable by every job on the controller, including a job somebody creates tomorrow.

Folder: team-payments
  └── credentials: payments-prod-deploy   ← only jobs in this folder
Folder: team-web
  └── credentials: web-staging-deploy

The rule that prevents the worst outcome: a build triggered by a fork’s pull request must not be able to reach any credential that can deploy, push or read production data. That build runs code you have not reviewed. Multibranch organisation folders let you build fork PRs without credentials, or only after a collaborator approves — and the default is more permissive than most teams realise.

Two related habits:

  • Never let a PR build deploy. PR builds compile, test and scan.
  • Separate read from write. A build that only needs to pull an image should not hold a push credential.

Topic 4: External Secret Stores

Storing secrets in Jenkins makes Jenkins the thing that must never be compromised. Keeping them elsewhere reduces that:

HashiCorp Vault — fetch at build time, with a short lease:

withVault(vaultSecrets: [[
    path: 'secret/data/payments/prod',
    secretValues: [[envVar: 'DB_PASSWORD', vaultKey: 'password']]
]]) {
    sh './migrate.sh'
}

Cloud secret managers — AWS Secrets Manager, GCP Secret Manager, Azure Key Vault, read using the build’s own cloud identity:

sh 'gcloud secrets versions access latest --secret=db-password > /tmp/pw'

The gain is not that the secret is more secret; it is that Jenkins holds one identity instead of fifty secrets, rotation happens at the source, and access is logged in the secret store’s audit trail rather than only in Jenkins.


Topic 5: OIDC — Removing the Secret Entirely

The strongest option is to have no stored cloud credential at all. Jenkins presents a signed identity token; the cloud exchanges it for short-lived credentials.

// GCP Workload Identity Federation from Jenkins on GKE
steps {
    sh '''
      gcloud auth login --cred-file=/var/run/secrets/gcp/credential-configuration.json
      gcloud run deploy checkout --image ${IMAGE}
    '''
}
// AWS, via the plugin that assumes a role with a web identity token
withAWS(roleAccount: '111122223333', role: 'jenkins-deploy', duration: 900) {
    sh 'aws s3 sync ./dist s3://assets/'
}

What this changes: there is no long-lived key to leak, rotate or find in a log. The credential lasts fifteen minutes and is scoped to one role. The trust is expressed in the cloud’s IAM — “this Jenkins service account, from this cluster, may assume this role” — which is auditable in a place Jenkins does not control.

This is the same mechanism the AWS and GCP modules cover from the cloud side (IRSA, Workload Identity, AssumeRoleWithWebIdentity), and it is the single largest improvement available to a Jenkins installation holding static cloud keys.


Topic 6: An Audit You Can Run Today

// A pipeline step that lists every credential ID the controller holds
script {
    def creds = com.cloudbees.plugins.credentials.CredentialsProvider.lookupCredentials(
        com.cloudbees.plugins.credentials.common.StandardCredentials.class,
        jenkins.model.Jenkins.instance, null, null)
    creds.each { echo "${it.id} — ${it.description} (${it.getClass().simpleName})" }
}

(That needs script approval, which is itself a reminder of how much a pipeline could do on the controller — see the operations lesson.)

The questions to answer, and they are usually uncomfortable:

□ How many credentials are global rather than folder-scoped?
□ Which are static cloud keys that could be OIDC instead?
□ Which have not been rotated in a year? In three?
□ Which jobs can reach a production deploy credential?
□ Can a fork pull request reach any credential at all?
□ Is any credential shared between staging and production?

The last one matters more than it looks: a shared credential means a staging compromise is a production compromise, and staging is always the less-defended environment.

Try it yourself: print a secret with sh "echo ${TOKEN}" and again with sh 'echo $TOKEN', and compare the build log and the pipeline step view. Seeing the value appear in one and not the other is what makes the quoting rule stick.

Common mistake: relying on Jenkins’ log masking as the control that keeps secrets safe. It matches exact strings in console output and nothing else — not transformed values, not the process list, not files, not artifacts, not a tool’s own error output. Treat every secret a build touches as potentially logged, and reduce the blast radius instead: short-lived tokens, narrow scope, and no production credential anywhere near untrusted code.