Testing, Secrets & Knowing When to Leave Bash

Shell scripts run in production with root privileges and nobody tests them. This lesson closes both gaps, and then makes the case for writing less shell.

Topic 1: Making a Script Testable

A script that executes on load cannot be tested — sourcing it runs it. The fix is the main guard:

#!/usr/bin/env bash
set -Eeuo pipefail

parse_duration() {
    local input=$1
    case $input in
        *s) echo "${input%s}" ;;
        *m) echo $(( ${input%m} * 60 )) ;;
        *h) echo $(( ${input%h} * 3600 )) ;;
        *)  echo "invalid duration: ${input}" >&2; return 1 ;;
    esac
}

main() {
    parse_duration "$1"
}

# Only run main when EXECUTED, not when SOURCED
if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
    main "$@"
fi

That final guard is the entire technique. Tests source the file to get its functions; running it directly still works exactly as before.

Design for testability the same way you would in any language:

• Small functions with one job. A function that reads a file, transforms it, and uploads it cannot be tested; three functions can.
• Return values via stdout, status via exit code. Not by setting globals.
• Push side effects to the edges. Pure logic in the middle is trivially testable.

Topic 2: bats

bats-core is the standard test framework for bash. A test file is a shell script with @test blocks.

# tests/test_duration.bats
setup() {
    load '../lib/duration.sh'          # source the code under test
}

@test "parses seconds" {
    run parse_duration "30s"
    [ "$status" -eq 0 ]
    [ "$output" = "30" ]
}

@test "converts minutes to seconds" {
    run parse_duration "5m"
    [ "$output" = "300" ]
}

@test "rejects an invalid unit" {
    run parse_duration "10x"
    [ "$status" -ne 0 ]
    [[ "$output" == *"invalid duration"* ]]
}

@test "handles a filename with spaces" {
    local f="${BATS_TEST_TMPDIR}/my report.txt"
    touch "$f"
    run process_file "$f"
    [ "$status" -eq 0 ]
}

bats tests/                 # run everything
bats -t tests/              # TAP output for CI
bats --filter duration tests/

run is the core helper: it executes a command, captures stdout+stderr into $output, the exit code into $status, and — importantly — stops a failure from aborting the test file. ${lines[0]} indexes individual output lines.

BATS_TEST_TMPDIR gives each test a fresh temporary directory that bats cleans up, which is where filename edge cases belong.

Mocking external commands:

The dependency problem: your script calls aws, kubectl, or curl, and tests must not.

setup() {
    # Put a fake bin directory FIRST on PATH
    export MOCK_BIN="${BATS_TEST_TMPDIR}/bin"
    mkdir -p "$MOCK_BIN"
    cat > "${MOCK_BIN}/curl" <<'EOF'
#!/usr/bin/env bash
echo '{"status":"ok"}'
EOF
    chmod +x "${MOCK_BIN}/curl"
    export PATH="${MOCK_BIN}:${PATH}"
}

@test "parses a healthy API response" {
    run check_health
    [ "$status" -eq 0 ]
}

Prepending a directory to PATH is the whole mocking mechanism, and it works for any external command.

What is worth testing:

Not everything. Test the logic — parsing, branching, computation, error paths. Do not write tests asserting that rsync copies files; that is rsync’s job. The highest-value tests are the edge cases from this module: a filename with a space, an empty variable, a command that returns non-zero.

Topic 3: Secrets

Shell leaks credentials in four places, and most scripts leak in at least one.

1. The process list — world-readable:

# WRONG: visible to every user on the box via `ps aux`
mysql -u root -p"${DB_PASSWORD}" -e 'SELECT 1'
curl -H "Authorization: Bearer ${TOKEN}" "$url"

# Right: a file, an env var the tool reads, or stdin
mysql --defaults-extra-file=<(printf '[client]\npassword=%s\n' "$DB_PASSWORD")
curl -H @<(printf 'Authorization: Bearer %s\n' "$TOKEN") "$url"
PGPASSWORD="$DB_PASSWORD" psql -c 'SELECT 1'      # env is not in ps output

2. set -x output:

Tracing prints every expanded command, secrets included, into whatever log CI keeps forever.

set +x                       # disable around the sensitive part
curl -H "Authorization: Bearer ${TOKEN}" "$url"
set -x

3. Shell history:

Any command typed with a secret lands in ~/.bash_history. A leading space omits it when HISTCONTROL=ignorespace is set — but that is a personal setting, not a guarantee.

4. Error messages and logs:

log "connecting with token ${TOKEN}"        # now it is in the log forever

Redact at the boundary:

redact() { sed -E 's/(token|password|secret)=[^ &]*/\1=REDACTED/gi'; }
run_thing 2>&1 | redact | tee -a "$LOG_FILE"

Where secrets should come from:

# From the environment, injected by the orchestrator
: "${API_TOKEN:?API_TOKEN is required}"

# From a file with tight permissions
readonly TOKEN=$(< /etc/myapp/token)          # verify: chmod 600, owned by the service user

# From a secret manager, fetched at runtime and never written to disk
TOKEN=$(aws secretsmanager get-secret-value --secret-id prod/api --query SecretString --output text)
TOKEN=$(vault kv get -field=token secret/prod/api)

Never commit a secret to the repository. git-secrets or gitleaks in a pre-commit hook catches the accident before it becomes a rotation exercise.

Topic 4: Other Security Habits

#!/usr/bin/env bash
set -Eeuo pipefail

# 1. Explicit PATH -- do not inherit a hostile one
export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

# 2. Restrictive umask for anything the script creates
umask 077

# 3. Validate input against an allow-list, never a deny-list
readonly ENVIRONMENT=$1
case $ENVIRONMENT in
    dev|staging|production) ;;
    *) echo "invalid environment: ${ENVIRONMENT}" >&2; exit 2 ;;
esac

# 4. Never eval user input
eval "$user_input"                    # arbitrary code execution. Never.

# 5. Guard destructive paths
readonly TARGET=${DEPLOY_DIR:?DEPLOY_DIR must be set}
[[ $TARGET == /srv/app/* ]] || { echo "refusing to touch ${TARGET}" >&2; exit 1; }
rm -rf "${TARGET:?}/"                 # :? aborts if somehow empty

The ${TARGET:?} inside rm -rf is a genuine last line of defence: if the variable is empty, the expansion errors and rm never runs. That is the difference between a failed deploy and an erased filesystem.

Two more worth internalising:

• Quote every path. The quoting lesson, applied where it can delete things.
• Prefer -- before user-controlled arguments: rm -- "$file" stops a file named -rf being read as flags.

Topic 5: Knowing When to Leave

The Google Shell Style Guide sets the threshold at ~100 lines, or any script doing non-trivial data manipulation. It is a good rule.

Signals it is time:

• You are simulating a data structure with parallel arrays or delimited strings.
• You are parsing anything with nesting — JSON beyond jq one-liners, XML, YAML.
• You need floating-point arithmetic.
• You need to distinguish more than a handful of error conditions.
• The script has grown past ~200 lines with several layers of functions.
• You are writing your own retry, config, and logging framework — all of which exist in Python’s standard library.

The hybrid that works:

Do not rewrite everything. Keep shell for what shell is superb at — process orchestration, pipes, environment — and delegate the logic:

#!/usr/bin/env bash
set -Eeuo pipefail
SCRIPT_DIR=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)

# Shell: gather, orchestrate, react
kubectl get pods -A -o json \
  | python3 "${SCRIPT_DIR}/../lib/find_unhealthy.py" \
  | while IFS=$'\t' read -r namespace pod reason; do
        echo "restarting ${pod} in ${namespace}: ${reason}"
        kubectl -n "$namespace" delete pod "$pod"
    done

Shell fetches and acts; Python does the analysis. Each language is doing what it is good at, and the boundary is a tab-separated stream — the oldest interface in Unix and still one of the best.

What Python buys at the boundary:

Try it yourself: Run wc -l across your scripts directory, sorted. Open the longest one and find the function doing the most data manipulation — porting that single function to Python, called from the same shell script, is usually a one-hour change with a large payoff.

Common mistake: Treating “rewrite it in Python” as an admission of failure and growing a 600-line bash script instead. Shell is glue. Reaching for a real language when the logic outgrows the glue is the senior judgement call, not a defeat.