Jenkins CI/CD: Pipelines, Agents & Deployments

Jenkins is the automation server that sits at the center of most self-hosted CI/CD setups: it watches your repositories, runs builds and tests the moment code changes, and — when everything passes — pushes that code out to servers, registries, or orchestrators. It is not the newest tool in the CI/CD space, but it remains the most flexible: a huge plugin ecosystem, first-class support for both GitHub and Bitbucket, and a pipeline model expressive enough to encode almost any real-world release process as code.

This guide covers standing up Jenkins from scratch, understanding the controller/agent execution model, configuring credentials and tool installations, writing declarative pipelines, wiring webhooks from GitHub and Bitbucket so builds trigger on every push, and deploying build output to a server or a container registry. It closes with the troubleshooting instincts and interview questions that come from running Jenkins in production rather than just reading about it.

Who this is for

• Engineers setting up CI/CD for a team that currently ships by hand.
• DevOps engineers who need to migrate ad-hoc deploy scripts into a real pipeline.
• Anyone who has used Jenkins as a black box and wants to understand what is actually happening underneath.

Before working through the installation and pipeline sections below, have ready:

What CI/CD actually buys you

Continuous Integration means every push is built and tested automatically, so integration bugs surface within minutes instead of at release time. Continuous Delivery extends that to producing a release-ready artifact on every successful build. Continuous Deployment goes one step further and pushes that artifact to production automatically, with no manual gate. Jenkins can drive any of the three — which one you get depends entirely on what your pipeline does after the tests pass.

The controller / agent model

A Jenkins deployment has one controller(formerly called the "master") that owns the UI, the job configuration, the plugin ecosystem, and the scheduling queue. Actual build work happens on agents(formerly "slaves") — separate machines or containers that connect to the controller and execute the steps of a pipeline. This split matters for two reasons: it keeps untrusted build workloads off the controller, and it lets you run builds with different OSes, tool versions, or hardware (a Windows agent for a .NET build, a GPU agent for ML jobs) from the same controller.

Declarative vs scripted pipeline syntax

Jenkins pipelines are written in Groovy DSL, checked into the repo as a Jenkinsfile. There are two syntaxes:

A declarative pipeline is built from stages (logical phases like Build, Test, Deploy, shown as columns in the Jenkins UI), each containing one or more steps (the actual commands). A post block at the pipeline or stage level runs regardless of outcome — the natural place for notifications and cleanup.

Static vs dynamic (cloud) agents

A static agentis a long-lived machine you register once and it stays connected, ready to pick up builds — simple, predictable, but idle capacity when there's no work. A dynamic/cloud agent is provisioned on demand (a Docker container, a Kubernetes pod, an EC2 instance) for a single build and then torn down — better utilization at the cost of slightly higher per-build startup time. The Kubernetes plugin and Docker plugin are the two most common ways to run dynamic agents.

A typical setup: source pushes from GitHub and Bitbucket trigger the controller via webhook, the controller queues the build and dispatches it to a labeled agent, and the agent does the actual work of building, testing, and shipping to a registry or server.

Jenkins ships three practical installation paths. All three end up at the same web UI on port 8080 — pick whichever fits how the rest of your infrastructure is run.

# Requires a supported JDK (17 or 21) already installed
java -version

# Download the latest LTS WAR
curl -fsSL -O https://get.jenkins.io/war-stable/latest/jenkins.war

# Run it directly (foreground, good for a quick trial)
java -jar jenkins.war --httpPort=8080

# Get the initial admin password
cat /root/.jenkins/secrets/initialAdminPassword

Credentials store

Manage Jenkins → Credentials is an encrypted store, never plain environment variables in a job config. Every credential has a type and an ID — the ID is what pipelines reference:

Pipelines pull credentials in with withCredentials or the shorthand environment { ... credentials('id') }block — Jenkins masks the value in the console log automatically as long as it's referenced this way rather than echoed manually:

withCredentials([
    usernamePassword(credentialsId: 'registry-creds', usernameVariable: 'REG_USER', passwordVariable: 'REG_PASS'),
    sshUserPrivateKey(credentialsId: 'deploy-ssh-key', keyFileVariable: 'SSH_KEY')
]) {
    sh '''
        echo "$REG_PASS" | docker login registry.example.com -u "$REG_USER" --password-stdin
        ssh -i "$SSH_KEY" -o StrictHostKeyChecking=no deploy@prod01 "systemctl restart app"
    '''
}

Global tool configuration

Manage Jenkins → Tools lets you register named installations of JDK, Maven, Gradle, Node.js, and Docker — either pointing at a fixed path or letting Jenkins auto-install them per agent on first use. Pipelines then reference tools by name instead of hardcoding paths:

pipeline {
    agent any
    tools {
        jdk 'jdk17'
        maven 'maven-3.9'
        nodejs 'node20'
    }
    stages {
        stage('Build') {
            steps {
                sh 'mvn -B -DskipTests package'
            }
        }
    }
}

Agents and nodes

Manage Jenkins → Nodes is where static agents are registered: give the node a name, one or more labels, a remote root directory, and a launch method — usually "Launch agent by connecting it to the controller" (JNLP/inbound, agent initiates the connection, works behind NAT) or "Launch via SSH" (controller connects out to the agent, needs the agent reachable and a credential registered).

# Download the matching agent.jar from the controller
curl -sO http://jenkins.example.com:8080/jnlpJars/agent.jar

# Connect using the secret shown on the Node's configuration page
java -jar agent.jar -jnlpUrl http://jenkins.example.com:8080/computer/agent-01/jenkins-agent.jnlp \
  -secret <secret-from-ui> -workDir "/home/jenkins/agent"

Jenkins CLI

# Download the CLI jar from a running controller
curl -sO http://jenkins.example.com:8080/jnlpJars/jenkins-cli.jar

# Authenticate with an API token (Manage Jenkins > Users > Security > API Token)
alias jcli='java -jar jenkins-cli.jar -s http://jenkins.example.com:8080/ -auth admin:$JENKINS_TOKEN'

jcli list-jobs                          # list all jobs
jcli build my-pipeline -f -v            # trigger a build, follow output
jcli console my-pipeline                # last build's console log
jcli get-job my-pipeline > config.xml   # export job config as XML
jcli create-job new-job < config.xml    # create a job from XML
jcli safe-restart                       # restart once builds finish
jcli quiet-down                         # stop accepting new builds
jcli list-plugins

Common Groovy pipeline snippets

// Read a value out of a properties/JSON/YAML file
def props = readProperties file: 'version.properties'
echo "Building version ${props.VERSION}"

// Conditional stage execution
stage('Deploy to prod') {
    when { branch 'main' }
    steps { sh './deploy.sh prod' }
}

// Parallel stages
stage('Test matrix') {
    parallel {
        stage('Unit')        { steps { sh 'npm test' } }
        stage('Integration') { steps { sh 'npm run test:int' } }
    }
}

// Retry a flaky step
retry(3) {
    sh 'curl -f https://staging.example.com/health'
}

// Timeout a hung step
timeout(time: 10, unit: 'MINUTES') {
    sh './long-running-migration.sh'
}

A complete declarative pipeline: checkout, install dependencies, run tests, build a Docker image, push it to a registry, then deploy over SSH — the shape almost every service-style app ends up using.

pipeline {
    agent { label 'linux-docker' }

    environment {
        IMAGE = "registry.example.com/acme/api"
        TAG   = "${env.GIT_COMMIT.take(7)}"
    }

    options {
        timestamps()
        buildDiscarder(logRotator(numToKeepStr: '20'))
        disableConcurrentBuilds()
    }

    stages {
        stage('Checkout') {
            steps {
                checkout scm
            }
        }

        stage('Install & Unit Test') {
            steps {
                sh 'npm ci'
                sh 'npm run lint'
                sh 'npm test -- --ci --reporters=default --reporters=jest-junit'
            }
            post {
                always {
                    junit 'reports/junit.xml'
                }
            }
        }

        stage('Build Image') {
            steps {
                sh "docker build -t ${IMAGE}:${TAG} -t ${IMAGE}:latest ."
            }
        }

        stage('Push Image') {
            steps {
                withCredentials([usernamePassword(
                    credentialsId: 'registry-creds',
                    usernameVariable: 'REG_USER',
                    passwordVariable: 'REG_PASS'
                )]) {
                    sh '''
                        echo "$REG_PASS" | docker login registry.example.com -u "$REG_USER" --password-stdin
                        docker push $IMAGE:$TAG
                        docker push $IMAGE:latest
                    '''
                }
            }
        }

        stage('Deploy') {
            when { branch 'main' }
            steps {
                sshagent(credentials: ['deploy-ssh-key']) {
                    sh """
                        ssh -o StrictHostKeyChecking=no deploy@prod01 \
                          "docker pull ${IMAGE}:${TAG} && \
                           docker service update --image ${IMAGE}:${TAG} acme_api"
                    """
                }
            }
        }
    }

    post {
        success {
            slackSend(channel: '#builds', message: "✅ ${env.JOB_NAME} #${env.BUILD_NUMBER} deployed ${TAG}")
        }
        failure {
            slackSend(channel: '#builds', message: "❌ ${env.JOB_NAME} #${env.BUILD_NUMBER} failed")
        }
    }
}

Scenario: a team wants every push to mainon their GitHub repo to automatically build, test, and deploy to production with zero manual steps — the classic "deploy on push" setup.

• Keep the Jenkinsfile in the application repo (pipeline-as-code), never build steps typed into the job configuration UI.
• Pin agent labels per stage instead of agent any once you have more than one agent type — it prevents a build silently landing on the wrong toolchain.
• Set buildDiscarder and a pipeline timeout on every job so a hung or forgotten job doesn't consume executors or disk forever.
• Use shared libraries (vars/*.groovy in a separate repo) once more than two or three Jenkinsfiles start repeating the same stages.
• Treat the controller as a critical, backed-up system — back up JENKINS_HOME (jobs, credentials, plugin configs) on a schedule, not just the application servers it deploys to.
• Prefer declarative syntax by default; drop to scripted only for logic declarative genuinely cannot express.

• Credential binding, not environment leakage — always inject secrets via withCredentials/credentials() so Jenkins masks them in console output; never echo or manually export a raw secret value.
• Matrix/role-based authorization — replace the default "logged-in users can do anything" with Project-based Matrix Authorization or Role-Based Access Control so only the right teams can view/build/configure specific jobs.
• Script approval sandbox — the Groovy sandbox blocks unapproved dangerous method calls in pipeline scripts; approve individual signatures deliberately under Manage Jenkins → In-process Script Approval rather than disabling the sandbox.
• Agent-to-controller trust — enable Agent → Controller Access Control so agents (which run less-trusted build code) can't call back into controller-only APIs; never run untrusted PR builds on an agent that also holds deploy credentials.
• Webhook validation — configure a shared secret on GitHub/Bitbucket webhooks and verify the signature so arbitrary requests to the webhook URL can't trigger builds.
• Keep plugins current — the plugin manager flags known CVEs; outdated plugins are the most common real-world Jenkins compromise vector.

Jenkins earns its place in a modern toolchain by being unglamorous but completely flexible: a controller that schedules work, agents that execute it in isolation, a credentials store that keeps secrets out of logs, and a declarative Jenkinsfile that turns your entire release process — build, test, package, deploy — into code that lives next to the application it ships. Wire GitHub or Bitbucket webhooks in and that whole process runs on every push with no manual trigger. The fundamentals here — pipeline structure, agent labeling, credential binding, and webhook-driven triggers — carry directly into any CI/CD system you touch afterward, Jenkins or not.

• Official Jenkins documentation and pipeline syntax reference — jenkins.io
• Jenkins plugin index — plugins.jenkins.io
• GitHub webhook and Apps documentation — docs.github.com
• Bitbucket webhooks and integrations documentation — support.atlassian.com
• Groovy language documentation (pipeline scripting foundation) — groovy-lang.org