Project 3: Zero-Downtime Deploy & Rollback

The deploy script is the one everybody writes and nobody reviews, right up until it takes production down at 5pm on a Friday. This one is built around a single idea: the cutover is one atomic operation, and it is reversible.

The Requirements

#	Requirement	Why
1	Atomic activation — no partial state	Users must never see a half-deployed tree
2	Keep N previous releases on disk	Rollback must be instant, not a re-download
3	Health check gates the cutover	Never route traffic to something unproven
4	Automatic rollback on failure	The 3am path must need no thinking
5	Pre-flight validation before touching anything	Fail before you break, not after
6	Deploy lock	Two concurrent deploys is a corrupted release
7	Audit trail of who deployed what, when	The first question of every incident
8	`--dry-run`	Trust is built by showing the plan first

Step 1: The Release Layout

Everything depends on this directory structure:

/srv/app/
├── releases/
│   ├── 20260805-141203-a3f9c1/     ← previous
│   ├── 20260805-153044-b7e2d8/     ← previous
│   └── 20260805-161522-c9a4f3/     ← new, fully staged
├── current -> releases/20260805-161522-c9a4f3     ← a SYMLINK
└── shared/
    ├── config/
    ├── uploads/
    └── logs/

Two properties make this work:

Activation is one mv. Replacing a symlink with mv -T is atomic at the kernel level — there is no instant at which current points at nothing. A rm followed by ln -s has a window, however small, where the path does not exist, and that window is where 502s come from.

Rollback is the same operation, backwards. The previous release is still on disk, complete. Rolling back is re-pointing a symlink, which takes milliseconds and cannot fail for lack of network.

Shared state lives outside releases. Uploads, logs and config are symlinked into each release, so they survive both deploy and rollback.

Step 2: Skeleton

#!/usr/bin/env bash
#
# deploy.sh — atomic release deployment with health-gated cutover.
#
set -Eeuo pipefail

readonly SCRIPT_NAME=${0##*/}
readonly VERSION="2.0.0"

readonly APP_ROOT="${APP_ROOT:-/srv/app}"
readonly RELEASES_DIR="${APP_ROOT}/releases"
readonly CURRENT_LINK="${APP_ROOT}/current"
readonly SHARED_DIR="${APP_ROOT}/shared"
readonly KEEP_RELEASES="${KEEP_RELEASES:-5}"
readonly HEALTH_URL="${HEALTH_URL:-http://localhost:8080/health}"
readonly HEALTH_TIMEOUT="${HEALTH_TIMEOUT:-60}"
readonly SERVICE_NAME="${SERVICE_NAME:-app}"
readonly AUDIT_LOG="${APP_ROOT}/deploy-audit.log"

readonly E_USAGE=2 E_PREFLIGHT=3 E_BUILD=4 E_HEALTH=5 E_ROLLBACK=6

DRY_RUN=0
NEW_RELEASE=""
PREVIOUS_RELEASE=""

log()  { printf '%s [%-5s] %s\n' "$(date -Iseconds)" "$1" "${*:2}" >&2; }
info() { log INFO "$@"; }
warn() { log WARN "$@"; }
die()  { log ERROR "${*:1:$#-1}"; exit "${!#}"; }

run() {
    if (( DRY_RUN )); then info "DRY-RUN: $*"; return 0; fi
    "$@"
}

audit() {
    printf '%s\t%s\t%s\t%s\n' \
        "$(date -Iseconds)" "${SUDO_USER:-${USER:-unknown}}" "$1" "${2:-}" >> "$AUDIT_LOG"
}

Step 3: Pre-flight

Everything that can be checked before making a change, is.

preflight() {
    info "running pre-flight checks"

    for cmd in rsync curl systemctl; do
        command -v "$cmd" >/dev/null || die "missing command: ${cmd}" "$E_PREFLIGHT"
    done

    [[ -d $APP_ROOT ]]   || die "APP_ROOT does not exist: ${APP_ROOT}" "$E_PREFLIGHT"
    [[ -w $APP_ROOT ]]   || die "APP_ROOT not writable: ${APP_ROOT}" "$E_PREFLIGHT"
    run mkdir -p "$RELEASES_DIR" "$SHARED_DIR"/{config,uploads,logs}

    # Disk headroom — a deploy that fills the disk breaks the running version too
    local free_mb
    free_mb=$(df -Pm "$APP_ROOT" | awk 'NR==2 {print $4}')
    (( free_mb >= 500 )) || die "only ${free_mb}MB free in ${APP_ROOT}" "$E_PREFLIGHT"

    # The current release must be healthy BEFORE we start
    if [[ -L $CURRENT_LINK ]]; then
        PREVIOUS_RELEASE=$(readlink -f "$CURRENT_LINK")
        info "current release: ${PREVIOUS_RELEASE##*/}"
        if ! health_check 5; then
            warn "the CURRENT release is already unhealthy — deploying anyway"
        fi
    else
        info "no current release; this is the first deploy"
    fi

    info "pre-flight passed"
}

Checking the current release’s health first is what stops the classic confusion where a deploy is blamed for an outage that started twenty minutes earlier.

Step 4: Staging a Release

stage_release() {
    local source=$1
    local stamp revision
    stamp=$(date +%Y%m%d-%H%M%S)
    revision=$(git -C "$source" rev-parse --short HEAD 2>/dev/null || echo "nogit")
    NEW_RELEASE="${RELEASES_DIR}/${stamp}-${revision}"

    info "staging ${source} → ${NEW_RELEASE##*/}"
    run mkdir -p "$NEW_RELEASE"

    run rsync -a --delete \
        --exclude='.git' --exclude='node_modules/.cache' --exclude='*.log' \
        "${source}/" "${NEW_RELEASE}/" \
        || die "rsync failed" "$E_BUILD"

    # Link shared state INTO the release
    local dir
    for dir in config uploads logs; do
        run rm -rf "${NEW_RELEASE:?}/${dir}"
        run ln -sfn "${SHARED_DIR}/${dir}" "${NEW_RELEASE}/${dir}"
    done

    # Anything that must happen before traffic arrives
    if [[ -x "${NEW_RELEASE}/bin/build" ]]; then
        info "running build hook"
        run "${NEW_RELEASE}/bin/build" || die "build hook failed" "$E_BUILD"
    fi

    printf '%s\n' "$revision" > "${NEW_RELEASE}/.revision" 2>/dev/null || true
    info "staged ${NEW_RELEASE##*/}"
}

Note "${NEW_RELEASE:?}/${dir}" in the rm -rf. If NEW_RELEASE were somehow empty, :? aborts rather than letting rm -rf /config run. On a destructive line inside a deploy script, that guard is not paranoia.

Step 5: Health Check

health_check() {
    local timeout="${1:-$HEALTH_TIMEOUT}"
    local deadline=$(( SECONDS + timeout ))
    local attempt=0 code

    info "waiting for health at ${HEALTH_URL} (timeout ${timeout}s)"
    while (( SECONDS < deadline )); do
        (( attempt++ )) || true
        code=$(curl -sS -o /dev/null -w '%{http_code}' --max-time 5 "$HEALTH_URL" 2>/dev/null || echo 000)

        if [[ $code == 200 ]]; then
            info "healthy after ${attempt} attempts"
            return 0
        fi
        sleep 2
    done

    warn "health check FAILED after ${timeout}s (last status ${code})"
    return 1
}

Using $SECONDS for the deadline rather than counting iterations means the timeout is honest even when individual curl calls take their full 5 seconds.

Step 6: Atomic Activation

activate() {
    local target=$1
    info "activating ${target##*/}"

    # mv -T on a symlink is ATOMIC — no window where `current` is missing
    run ln -sfn "$target" "${CURRENT_LINK}.tmp"
    run mv -Tf "${CURRENT_LINK}.tmp" "$CURRENT_LINK"

    if systemctl list-units --type=service --all 2>/dev/null | grep -q "${SERVICE_NAME}.service"; then
        info "reloading ${SERVICE_NAME}"
        run systemctl reload-or-restart "$SERVICE_NAME" || die "service reload failed" "$E_HEALTH"
    fi
}

rollback_to() {
    local target=$1
    warn "ROLLING BACK to ${target##*/}"
    activate "$target"
    if health_check 30; then
        warn "rollback complete and healthy"
        audit "rollback-success" "${target##*/}"
        return 0
    fi
    log ERROR "ROLLBACK FAILED — the previous release is also unhealthy. Manual intervention required."
    audit "rollback-failed" "${target##*/}"
    return 1
}

ln -sfn to a temporary name followed by mv -Tf is the atomic-symlink-swap idiom. -T treats the destination as a file rather than descending into it — without it, mv would move the link inside the existing current directory.

Step 7: The Deploy Flow

cmd_deploy() {
    local source="."
    local OPTIND=1
    while getopts ":s:h" opt; do
        case $opt in
            s) source=$OPTARG ;;
            h) echo "usage: ${SCRIPT_NAME} deploy [-s SOURCE]"; return 0 ;;
            \?) die "deploy: unknown option -${OPTARG}" "$E_USAGE" ;;
        esac
    done

    [[ -d $source ]] || die "source not found: ${source}" "$E_USAGE"

    exec 200>"${APP_ROOT}/.deploy.lock"
    flock -n 200 || die "another deploy is in progress" "$E_USAGE"

    audit "deploy-start" "$source"
    preflight
    stage_release "$source"
    activate "$NEW_RELEASE"

    if health_check; then
        info "DEPLOY SUCCESSFUL: ${NEW_RELEASE##*/}"
        audit "deploy-success" "${NEW_RELEASE##*/}"
        prune_releases
        return 0
    fi

    # The whole point of the project
    warn "new release is unhealthy — initiating automatic rollback"
    audit "deploy-failed" "${NEW_RELEASE##*/}"

    if [[ -z $PREVIOUS_RELEASE ]]; then
        die "no previous release to roll back to" "$E_HEALTH"
    fi

    rollback_to "$PREVIOUS_RELEASE" || exit "$E_ROLLBACK"
    exit "$E_HEALTH"
}

cmd_rollback() {
    [[ -L $CURRENT_LINK ]] || die "nothing is currently deployed" "$E_USAGE"
    local current; current=$(readlink -f "$CURRENT_LINK")

    # The newest release that is NOT the current one
    local -a releases
    mapfile -t releases < <(find "$RELEASES_DIR" -maxdepth 1 -mindepth 1 -type d | sort -r)

    local target=""
    local r
    for r in "${releases[@]}"; do
        [[ $r == "$current" ]] && continue
        target=$r
        break
    done

    [[ -n $target ]] || die "no previous release available" "$E_USAGE"
    audit "manual-rollback" "${target##*/}"
    rollback_to "$target" || exit "$E_ROLLBACK"
}

prune_releases() {
    local -a releases
    mapfile -t releases < <(find "$RELEASES_DIR" -maxdepth 1 -mindepth 1 -type d | sort -r)
    (( ${#releases[@]} <= KEEP_RELEASES )) && return 0

    local current=""
    [[ -L $CURRENT_LINK ]] && current=$(readlink -f "$CURRENT_LINK")

    local i
    for (( i = KEEP_RELEASES; i < ${#releases[@]}; i++ )); do
        [[ ${releases[i]} == "$current" ]] && continue      # never delete what is live
        info "pruning ${releases[i]##*/}"
        run rm -rf "${releases[i]:?}"
    done
}

cmd_status() {
    if [[ -L $CURRENT_LINK ]]; then
        local c; c=$(readlink -f "$CURRENT_LINK")
        printf 'current:  %s\n' "${c##*/}"
        printf 'health:   %s\n' "$(health_check 5 >/dev/null 2>&1 && echo healthy || echo UNHEALTHY)"
    else
        echo "current:  (nothing deployed)"
    fi
    echo
    echo "releases on disk:"
    find "$RELEASES_DIR" -maxdepth 1 -mindepth 1 -type d -printf '  %f\n' 2>/dev/null | sort -r
    echo
    echo "recent deploys:"
    tail -5 "$AUDIT_LOG" 2>/dev/null || echo "  (no audit log)"
}

Step 8: Dispatch and CI

main() {
    while getopts ":nh" opt; do
        case $opt in
            n) DRY_RUN=1 ;;
            h) usage; exit 0 ;;
            \?) die "unknown option -${OPTARG}" "$E_USAGE" ;;
        esac
    done
    shift $(( OPTIND - 1 ))

    (( $# > 0 )) || { usage >&2; exit "$E_USAGE"; }
    local sub=$1; shift
    case $sub in
        deploy|rollback|status) "cmd_${sub}" "$@" ;;
        help) usage; exit 0 ;;
        *) die "unknown command: ${sub}" "$E_USAGE" ;;
    esac
}

if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
    main "$@"
fi

# .github/workflows/deploy.yml
- name: Deploy
  run: |
    rsync -az --delete ./dist/ deploy@prod:/tmp/incoming/
    ssh -n deploy@prod '/usr/local/bin/deploy.sh deploy -s /tmp/incoming'

Exit code 5 (health failure, rolled back) is distinct from 6 (rollback itself failed) — so CI can mark the first as a failed deploy and the second as a page someone immediately.

Verifying Your Work

export APP_ROOT=/tmp/deploytest HEALTH_URL=http://localhost:8080/health

# 1. First deploy
mkdir -p /tmp/src && echo v1 > /tmp/src/version.txt
./deploy.sh deploy -s /tmp/src
cat /tmp/deploytest/current/version.txt        # v1

# 2. Second deploy — current must switch atomically
echo v2 > /tmp/src/version.txt
./deploy.sh deploy -s /tmp/src
cat /tmp/deploytest/current/version.txt        # v2

# 3. THE test: deploy something that fails its health check
#    (point HEALTH_URL at a dead port)
HEALTH_URL=http://localhost:9999/health HEALTH_TIMEOUT=10 ./deploy.sh deploy -s /tmp/src
cat /tmp/deploytest/current/version.txt        # still the last GOOD release

# 4. Manual rollback
./deploy.sh rollback
./deploy.sh status

# 5. Concurrency
./deploy.sh deploy -s /tmp/src & ./deploy.sh deploy -s /tmp/src; wait

# 6. Retention
for i in {1..8}; do echo "v$i" > /tmp/src/version.txt; ./deploy.sh deploy -s /tmp/src; done
ls /tmp/deploytest/releases | wc -l            # KEEP_RELEASES

Step 3 is the project. Everything else is scaffolding around the requirement that a bad release cannot stay live.

Extensions worth building: blue/green with two symlinks and a load-balancer flip; database migrations with a paired down step run on rollback; canary — activate for 10% of traffic and compare error rates before full cutover; a Slack notification on deploy start, success and rollback using the alerting project’s notify function.

The lesson to take away: rollback is not an error path bolted on at the end — it is the primary feature. Design the layout so that reverting is the cheapest operation in the system, and a bad deploy stops being an incident and becomes a non-event.