Jenkins is a Java application that schedules work and stores everything it knows in one directory. Almost every operational surprise — backups, upgrades, security, capacity — follows from those two facts.
Topic 1: Controller and Agents
The controller serves the UI, stores job configuration and build history, schedules builds, and holds credentials. It is a single point of failure and, in most installations, a single point of compromise.
Agents are machines or containers that execute the work. Each has some number of executors — concurrent build slots. An agent connects to the controller over SSH (controller-initiated) or JNLP/WebSocket (agent-initiated, which is what works through a firewall or from inside Kubernetes).
Set the controller’s executor count to zero. This is the first configuration change on any new Jenkins, and it is a security control rather than a performance one: a build running on the controller shares a filesystem with secrets/, credentials.xml and every job’s configuration. One malicious or careless build step reads everything Jenkins knows.
// JCasC — the setting worth doing declaratively from day one
jenkins:
numExecutors: 0
mode: EXCLUSIVE // only jobs that explicitly ask for the controller
Topic 2: JENKINS_HOME Is the Whole System
/var/jenkins_home/
├── config.xml global configuration
├── jenkins.yaml JCasC, if you use it (and you should)
├── credentials.xml encrypted with…
├── secrets/ …the keys in here. Both, or neither.
├── jobs/
│ └── my-job/
│ ├── config.xml the job definition
│ └── builds/ history, logs, artifacts — this is the bulk
├── plugins/ .jpi files, and your largest risk surface
├── nodes/ agent definitions
├── workspace/ scratch; safe to lose
└── users/
Three operational consequences:
Backups are simple and specific. config.xml, jobs/*/config.xml, credentials.xml, secrets/, plugins/*.jpi, users/, nodes/. You can skip workspace/ and, if you accept losing history, jobs/*/builds/ — which is usually 90% of the size.
tar czf jenkins-backup-$(date +%F).tgz \
-C /var/jenkins_home \
config.xml jenkins.yaml credentials.xml secrets users nodes \
--exclude='jobs/*/builds' jobs plugins
credentials.xml without secrets/ is useless, and secrets/ without credentials.xml is useless. Back up both, and treat the pair as production-secret material — anyone with both has every credential Jenkins holds.
Build history is the thing that grows. A job retaining every build forever fills the disk, and a full JENKINS_HOME disk makes Jenkins fail in confusing ways. Set buildDiscarder in every pipeline:
options {
buildDiscarder(logRotator(numToKeepStr: '30', artifactNumToKeepStr: '5'))
}
Topic 3: Installing It Sensibly
docker run -d --name jenkins \
-p 8080:8080 -p 50000:50000 \
-v jenkins_home:/var/jenkins_home \
-e JAVA_OPTS="-Djenkins.install.runSetupWizard=false" \
-e CASC_JENKINS_CONFIG=/var/jenkins_home/jenkins.yaml \
jenkins/jenkins:lts-jdk17
Port 8080 is the UI; 50000 is the JNLP agent port, needed only if agents connect inbound. Do not expose it to the internet.
# The initial admin password, on a first run without JCasC
docker exec jenkins cat /var/jenkins_home/secrets/initialAdminPassword
Choices worth making deliberately at install time:
- Use the
ltstag, pinned to a specific version. Jenkins ships weekly releases and quarterly LTS lines; production should track LTS and update on a schedule, not on a whim. - Install plugins from a
plugins.txt, not by clicking. This is what makes the controller reproducible:
# plugins.txt
workflow-aggregator:latest
git:latest
configuration-as-code:latest
kubernetes:latest
credentials-binding:latest
FROM jenkins/jenkins:2.462.3-lts-jdk17
COPY plugins.txt /usr/share/jenkins/ref/plugins.txt
RUN jenkins-plugin-cli -f /usr/share/jenkins/ref/plugins.txt
COPY jenkins.yaml /var/jenkins_home/jenkins.yaml
- Put it behind TLS, always. Jenkins carries credentials to everything; plaintext HTTP to it is a credential-harvesting opportunity on your own network.
- Do not run it as root, and do not mount the Docker socket into the controller.
On Kubernetes, the official Helm chart plus the Kubernetes plugin is the standard install, and it gives you dynamic agents for free — covered in the agents lesson.
Topic 4: The Plugin Problem, Stated Honestly
Jenkins’ capability comes from around 1,900 plugins, and so does most of its operational pain:
- Plugins are third-party code running with full access to your controller — and therefore to every credential it holds.
- Plugin CVEs are frequent, and the security advisories are worth subscribing to rather than discovering.
- Plugins have interdependencies, and an upgrade can break a job in a way that is only visible at build time.
- Abandoned plugins are common; a plugin whose last release was in 2019 is a liability you are carrying.
The discipline that keeps this manageable:
1. Install the fewest plugins that do the job. Audit annually and remove.
2. Pin versions in plugins.txt, upgrade deliberately, read the changelogs.
3. Stage upgrades: a non-production controller first, with real jobs.
4. Subscribe to the Jenkins security advisories.
5. Prefer a step that runs a container over a plugin that wraps a tool —
the container is versioned with your pipeline and has no controller access.
That last line is the strategic one, and it is why the rest of this module leans on sh steps running pinned images rather than on plugins wherever there is a choice.
Topic 5: Security Baseline
The settings that matter, in the order an attacker would find them missing:
| Setting | Why |
|---|---|
| Authentication required, no anonymous read | An open Jenkins leaks source, credentials metadata and internal hostnames |
| Matrix or role-based authorisation | ”Logged-in users can do anything” means any account is an admin |
| Controller executors = 0 | Builds cannot read secrets/ |
| Agent-to-controller access control on | Stops an agent from calling controller-side APIs |
| Script security / sandbox enabled | Unsandboxed Groovy in a job is arbitrary code on the controller |
| CSRF protection on (default) | Leave it |
| Credentials scoped to folders | A job in one team cannot use another team’s deploy key |
| TLS, and the UI not on the public internet | Everything above assumes the network is not the front door |
The specific rule for pull-request builds: a build triggered by a fork’s pull request runs untrusted code. It must not have access to any credential that can deploy, push, or reach production. Multibranch pipelines can be configured to build PRs from forks without credentials, and that configuration is the difference between a CI system and an open shell on your infrastructure.
Topic 6: Sizing and When to Stop Scaling Up
Rough starting points, to be replaced by measurement:
controller 4 vCPU, 8–16 GB RAM, fast disk (JENKINS_HOME is I/O heavy)
JVM heap ~50% of RAM, and watch GC pauses
executors ~1–2 per agent vCPU for typical build work
agents scale horizontally; ephemeral if possible
The controller is single-threaded in places and does not scale by adding CPU. When one controller struggles, the answers in order are: move all builds off it (executors = 0), reduce build history retention, prune plugins, then split into several controllers by team or domain.
A controller with 2,000 jobs and 40,000 retained builds is slow in ways no amount of hardware fixes, because the bottleneck is the number of files it walks and the number of objects it holds in memory.
Try it yourself: run a build with the controller’s executor count at 1 and print sh 'ls -la /var/jenkins_home/secrets'. Then set executors to 0, attach an agent, and try again. Watching the first one succeed is the most convincing security argument in this lesson.
Common mistake: treating JENKINS_HOME as something to back up and restore rather than something to rebuild. A backup restores yesterday’s controller including yesterday’s manual UI changes that nobody documented. A JCasC file plus plugins.txt plus job definitions in Git rebuilds a controller you can reason about — and the backup then only has to carry build history and credentials.