Project 1: Backup & Restore System

An untested backup is not a backup. This project builds one that verifies itself, rotates on both count and age, refuses to run twice, and — critically — has a restore path you exercise before you need it.

The Requirements

Step 1: Skeleton and Configuration

#!/usr/bin/env bash
#
# backup.sh — create, verify and restore compressed backups.
#
set -Eeuo pipefail

readonly SCRIPT_NAME=${0##*/}
readonly SCRIPT_DIR=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)
readonly VERSION="1.0.0"

# Exit codes — distinct so callers can branch
readonly E_USAGE=2 E_CONFIG=3 E_DISK=4 E_INTEGRITY=5 E_LOCKED=0

# Defaults; every one overridable by environment then flag
SOURCE_DIR="${BACKUP_SOURCE:-/var/www}"
DEST_DIR="${BACKUP_DEST:-/backup}"
KEEP_COUNT="${BACKUP_KEEP:-7}"
KEEP_DAYS="${BACKUP_KEEP_DAYS:-30}"
MIN_FREE_MB="${BACKUP_MIN_FREE:-1024}"
EXCLUDE_FILE="${SCRIPT_DIR}/backup.exclude"
DRY_RUN=0
VERBOSE=0

E_LOCKED=0 is deliberate: another instance already running is expected behaviour under cron, not a failure worth mailing about.

Step 2: Logging and Guards

log() {
    local level=$1; shift
    printf '%s [%-5s] %s\n' "$(date -Iseconds)" "$level" "$*" >&2
}
info() { log INFO "$@"; }
warn() { log WARN "$@"; }
die()  { log ERROR "$@"; exit "${2:-1}"; }
debug() { (( VERBOSE )) && log DEBUG "$@"; return 0; }

run() {
    if (( DRY_RUN )); then
        info "DRY-RUN: $*"
        return 0
    fi
    "$@"
}

The run wrapper is the pattern that makes --dry-run trustworthy: every destructive call goes through it, so there is exactly one place where the guard can be wrong.

preflight() {
    [[ -d $SOURCE_DIR ]] || die "source not found: ${SOURCE_DIR}" "$E_CONFIG"
    [[ -r $SOURCE_DIR ]] || die "source not readable: ${SOURCE_DIR}" "$E_CONFIG"
    mkdir -p "$DEST_DIR"  || die "cannot create dest: ${DEST_DIR}" "$E_CONFIG"

    for cmd in tar gzip sha256sum find; do
        command -v "$cmd" >/dev/null || die "missing required command: ${cmd}" "$E_CONFIG"
    done

    local free_mb
    free_mb=$(df -Pm "$DEST_DIR" | awk 'NR==2 {print $4}')
    (( free_mb >= MIN_FREE_MB )) \
        || die "only ${free_mb}MB free in ${DEST_DIR}, need ${MIN_FREE_MB}MB" "$E_DISK"

    debug "preflight passed: ${free_mb}MB free"
}

Checking free space before starting is what stops a backup filling the disk it is protecting and taking the host down with it.

Step 3: Locking and Cleanup

readonly LOCKFILE="/var/lock/${SCRIPT_NAME%.sh}.lock"
WORKDIR=""

cleanup() {
    local code=$?
    [[ -n $WORKDIR && -d $WORKDIR ]] && rm -rf "$WORKDIR"
    (( code != 0 )) && log ERROR "exiting with status ${code}"
    return "$code"
}
trap cleanup EXIT
trap 'exit 130' INT
trap 'exit 143' TERM

acquire_lock() {
    exec 200>"$LOCKFILE"
    flock -n 200 || { info "another instance holds the lock; skipping"; exit "$E_LOCKED"; }
    debug "lock acquired"
}

Step 4: Creating the Backup

cmd_create() {
    preflight
    acquire_lock

    local stamp archive checksum
    stamp=$(date +%Y%m%d-%H%M%S)
    archive="${DEST_DIR}/backup-${stamp}.tar.gz"
    checksum="${archive}.sha256"

    local -a tar_opts=(--create --gzip --file "$archive")
    [[ -r $EXCLUDE_FILE ]] && tar_opts+=(--exclude-from "$EXCLUDE_FILE")
    (( VERBOSE )) && tar_opts+=(--verbose)

    info "backing up ${SOURCE_DIR} → ${archive}"

    # -C so paths inside the archive are RELATIVE — restores anywhere
    if ! run tar "${tar_opts[@]}" -C "$(dirname "$SOURCE_DIR")" "$(basename "$SOURCE_DIR")"; then
        rm -f "$archive"
        die "tar failed; partial archive removed" "$E_DISK"
    fi

    if (( ! DRY_RUN )); then
        sha256sum "$archive" > "$checksum"
        local size
        size=$(du -h "$archive" | cut -f1)
        info "created ${archive##*/} (${size})"
        verify_archive "$archive" || die "verification failed immediately after creation" "$E_INTEGRITY"
    fi

    prune
}

Two details worth copying: building tar_opts as an array so an exclude path with a space survives, and using -C so the archive holds relative paths. An archive of /var/www with absolute paths can only ever be restored to /var/www.

Step 5: Verification

verify_archive() {
    local archive=$1
    local checksum="${archive}.sha256"

    [[ -f $archive ]] || { warn "missing archive: ${archive}"; return 1; }

    if [[ -f $checksum ]]; then
        if ! (cd "$(dirname "$archive")" && sha256sum --check --status "$(basename "$checksum")"); then
            warn "CHECKSUM MISMATCH: ${archive}"
            return 1
        fi
        debug "checksum ok: ${archive##*/}"
    else
        warn "no checksum file for ${archive##*/}"
    fi

    # Does tar consider the archive readable end to end?
    if ! tar --test-label --file "$archive" >/dev/null 2>&1 && ! tar -tzf "$archive" >/dev/null 2>&1; then
        warn "archive is unreadable: ${archive}"
        return 1
    fi
    return 0
}

cmd_verify() {
    local failed=0 checked=0
    shopt -s nullglob
    for archive in "${DEST_DIR}"/backup-*.tar.gz; do
        (( checked++ )) || true
        verify_archive "$archive" || (( failed++ )) || true
    done
    info "verified ${checked} archives, ${failed} failed"
    (( failed == 0 )) || exit "$E_INTEGRITY"
}

tar -tzf reads the entire archive, so it detects truncation and corruption that a checksum on a partially written file would not.

Step 6: Retention

prune() {
    shopt -s nullglob
    local -a archives=("${DEST_DIR}"/backup-*.tar.gz)
    (( ${#archives[@]} == 0 )) && return 0

    # Newest first — the timestamp format sorts lexically, so no stat needed
    mapfile -t archives < <(printf '%s\n' "${archives[@]}" | sort -r)

    local cutoff removed=0
    cutoff=$(date -d "${KEEP_DAYS} days ago" +%s 2>/dev/null || date -v-"${KEEP_DAYS}"d +%s)

    local i mtime
    for (( i = 0; i < ${#archives[@]}; i++ )); do
        (( i < KEEP_COUNT )) && continue          # always keep the N newest
        mtime=$(stat -c %Y "${archives[i]}" 2>/dev/null || stat -f %m "${archives[i]}")
        (( mtime > cutoff )) && continue          # keep anything inside the age window

        info "pruning $(basename "${archives[i]}")"
        run rm -f "${archives[i]}" "${archives[i]}.sha256"
        (( removed++ )) || true
    done
    (( removed > 0 )) && info "pruned ${removed} archives"
    return 0
}

The policy is count OR age — keep the 7 newest and anything under 30 days. A count-only policy loses a month of history after a week of hourly runs; an age-only policy leaves you with nothing if backups stop for 31 days.

The date -d / date -v pair handles GNU and BSD, so the script works on Linux and macOS.

Step 7: Restore — the Part Everyone Skips

cmd_restore() {
    local archive="" target="" force=0
    local OPTIND=1
    while getopts ":a:t:fh" opt; do
        case $opt in
            a) archive=$OPTARG ;;
            t) target=$OPTARG ;;
            f) force=1 ;;
            h) echo "usage: ${SCRIPT_NAME} restore -a ARCHIVE -t TARGET [-f]"; return 0 ;;
            \?) die "restore: unknown option -${OPTARG}" "$E_USAGE" ;;
        esac
    done

    [[ -n $archive ]] || die "restore: -a ARCHIVE is required" "$E_USAGE"
    [[ -n $target  ]] || die "restore: -t TARGET is required" "$E_USAGE"
    [[ -f $archive ]] || die "no such archive: ${archive}" "$E_CONFIG"

    verify_archive "$archive" || die "refusing to restore a corrupt archive" "$E_INTEGRITY"

    if [[ -d $target && -n $(ls -A "$target" 2>/dev/null) ]] && (( ! force )); then
        die "target ${target} is not empty; pass -f to overwrite" "$E_CONFIG"
    fi

    run mkdir -p "$target"
    info "restoring ${archive##*/} → ${target}"
    run tar --extract --gzip --file "$archive" -C "$target"
    info "restore complete"
}

The non-empty-target guard is the one that saves you. A restore that silently merges into an existing directory produces a tree that is neither the backup nor the original, and nobody notices until much later.

Step 8: Dispatch

usage() {
    cat <<EOF
${SCRIPT_NAME} ${VERSION} — backup, verify and restore

USAGE
    ${SCRIPT_NAME} [-n] [-v] <command> [options]

COMMANDS
    create              create a new backup and prune old ones
    list                list available backups with sizes
    verify              checksum and read-test every archive
    restore -a A -t T   restore archive A into directory T

GLOBAL OPTIONS
    -n    dry run
    -v    verbose
    -h    this help

ENVIRONMENT
    BACKUP_SOURCE, BACKUP_DEST, BACKUP_KEEP, BACKUP_KEEP_DAYS, BACKUP_MIN_FREE

EXIT STATUS
    0 ok (or skipped due to lock)   2 usage   3 config   4 disk   5 integrity
EOF
}

cmd_list() {
    shopt -s nullglob
    local -a archives=("${DEST_DIR}"/backup-*.tar.gz)
    (( ${#archives[@]} == 0 )) && { info "no backups in ${DEST_DIR}"; return 0; }
    printf '%-34s %8s  %s\n' ARCHIVE SIZE VERIFIED
    local a
    for a in "${archives[@]}"; do
        printf '%-34s %8s  %s\n' "$(basename "$a")" "$(du -h "$a" | cut -f1)" \
            "$(verify_archive "$a" >/dev/null 2>&1 && echo ok || echo FAILED)"
    done
}

main() {
    while getopts ":nvh" opt; do
        case $opt in
            n) DRY_RUN=1 ;;
            v) VERBOSE=1 ;;
            h) usage; exit 0 ;;
            \?) die "unknown option -${OPTARG}" "$E_USAGE" ;;
        esac
    done
    shift $(( OPTIND - 1 ))

    (( $# > 0 )) || { usage >&2; exit "$E_USAGE"; }
    local subcommand=$1; shift

    case $subcommand in
        create|list|verify|restore) "cmd_${subcommand}" "$@" ;;
        help) usage; exit 0 ;;
        *) die "unknown command: ${subcommand}" "$E_USAGE" ;;
    esac
}

if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
    main "$@"
fi

Step 9: Schedule It

# /etc/cron.d/backup
SHELL=/bin/bash
PATH=/usr/local/bin:/usr/bin:/bin
MAILTO=ops@example.com

30 2 * * * root /usr/bin/timeout 3600 /usr/local/bin/backup.sh create >> /var/log/backup.log 2>&1
0  6 * * 0 root /usr/local/bin/backup.sh verify >> /var/log/backup.log 2>&1

timeout bounds the run, the weekly verify catches silent corruption, and flock inside the script stops overlap.

Verifying Your Work

The project is not finished until this passes:

# 1. Create
./backup.sh -v create

# 2. Restore to a scratch directory
./backup.sh restore -a /backup/backup-*.tar.gz -t /tmp/restore-test

# 3. Prove it is byte-identical — THE test that matters
diff -r /var/www /tmp/restore-test/www && echo "RESTORE VERIFIED"

# 4. Retention behaves
for i in {1..10}; do ./backup.sh create; sleep 1; done
./backup.sh list        # should show KEEP_COUNT archives

# 5. Locking works
./backup.sh create & ./backup.sh create; wait     # second exits 0 with a skip message

# 6. Corruption is detected
echo garbage >> /backup/backup-*.tar.gz
./backup.sh verify      # must exit 5

Extensions worth building: encrypt with gpg --symmetric before upload; push to S3 with aws s3 cp and lifecycle rules; add a --incremental mode using tar --listed-incremental; emit a Slack notification on failure using the alerting lesson’s function.

The lesson to take away: step 3 of the verification is the entire point. A backup system that has never been restored from is a filesystem that has never been read. Schedule a restore test the same way you schedule the backup.