Modular Scripts, Libraries & Project Layout

Every useful script grows. The transition from one file to several is where shell projects usually become unmaintainable, because bash has no module system — only source, which is a blunt instrument.

Topic 1: `source` Is Not `import`

source lib/common.sh      # bash spelling
. lib/common.sh           # POSIX spelling -- identical behaviour

source reads the file and executes every line in the current shell. There is no namespace, no export list, no isolation. Everything the file defines — functions, variables, aliases, set options, even a stray cd — lands in your shell.

Two consequences drive the rest of this lesson:

A library must define only what it intends to, because everything leaks.
A library must not do anything on load, because loading is execution.

Sourcing versus executing:

./script.sh        # NEW process. Changes to variables/cwd do not affect you.
source script.sh   # CURRENT process. Everything it does happens to you.

This is why cd in a script does not change your shell’s directory, and why activation scripts (source venv/bin/activate) must be sourced rather than run.

Topic 2: Finding the Script’s Own Directory

A script cannot assume the working directory. ./bin/tool.sh and cd bin && ./tool.sh produce different $PWD for identical code, and cron runs everything from $HOME.

# The reliable idiom
SCRIPT_DIR=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)
readonly SCRIPT_DIR

source "${SCRIPT_DIR}/../lib/common.sh"

Why each piece:

${BASH_SOURCE[0]} rather than $0. When a file is sourced, $0 is the name of the parent shell, not the file — so $0 breaks in exactly the libraries that need this most.
dirname strips the filename.
cd ... && pwd resolves relative paths and symlinks in the parent chain to an absolute path.
-- stops option parsing, so a path beginning with - is not read as a flag.

To resolve symlinks to the script file itself, add readlink:

SCRIPT_PATH=$(readlink -f -- "${BASH_SOURCE[0]}")     # GNU only
SCRIPT_DIR=${SCRIPT_PATH%/*}

readlink -f is not on macOS by default. If portability matters, either accept the unresolved path or ship a small loop; do not assume -f exists.

Topic 3: A Layout That Scales

myproject/
├── bin/
│   └── deploy              # executable entry point, no extension
├── lib/
│   ├── common.sh           # logging, die, retry
│   ├── aws.sh              # one domain per file
│   └── validate.sh
├── etc/
│   └── defaults.conf       # sourceable KEY=value config
├── tests/
│   └── test_common.bats
└── README.md

Conventions worth following:

Entry points in bin/, no .sh extension. Users type deploy, not deploy.sh, and dropping the extension means you can rewrite it in Python later without changing every caller.
Libraries keep .sh and are not executable — they are meant to be sourced, and a non-executable bit documents that.
One domain per library. aws.sh, k8s.sh, notify.sh. When a file needs a table of contents, split it.

Topic 4: Writing a Library That Behaves

#!/usr/bin/env bash
# lib/common.sh — shared helpers. Source, do not execute.

# 1. Guard against double-sourcing
[[ -n "${_COMMON_SH_LOADED:-}" ]] && return 0
readonly _COMMON_SH_LOADED=1

# 2. Refuse to be executed directly
if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
    echo "common.sh is a library; source it instead" >&2
    exit 1
fi

# 3. Namespace everything
common::log()  { printf '%s [%s] %s\n' "$(date -Iseconds)" "$1" "${*:2}" >&2; }
common::info() { common::log INFO "$@"; }
common::die()  { common::log ERROR "$@"; exit 1; }

common::retry() {
    local attempts=$1 delay=$2; shift 2
    local n=1
    until "$@"; do
        (( n >= attempts )) && return 1
        common::info "attempt ${n}/${attempts} failed, retrying in ${delay}s"
        sleep "$delay"
        (( n++ )) || true
    done
}

The three guards, and why each earns its line:

1. The double-source guard. Libraries source other libraries, so a file can easily be loaded twice. Without the guard, readonly declarations fail on the second pass with “readonly variable” and — under set -e — kill the script.

2. The execution guard. ${BASH_SOURCE[0]} == $0 is true only when the file is run rather than sourced. It turns a confusing no-op into a clear message.

3. Namespacing. Bash has no modules, so log() in two libraries silently overwrites. The :: convention (common::log) is legal in bash function names and makes collisions impossible. Prefix internal-only helpers with _.

Variables need the same discipline:

readonly COMMON_DEFAULT_TIMEOUT=30       # exported constants: prefixed, readonly
_common_internal_state=""                # internals: leading underscore

Topic 5: Configuration

The tempting approach is to source a config file. That is fine for files you control and dangerous for anything else — sourcing executes, so a config containing rm -rf / runs it.

# Acceptable: a config file owned by root, in /etc, that you ship
source /etc/myapp/defaults.conf

For anything user-supplied, parse rather than execute:

while IFS='=' read -r key value; do
    [[ $key =~ ^[[:space:]]*# ]] && continue        # comments
    [[ -z $key ]] && continue                        # blanks
    case $key in
        TIMEOUT|RETRIES|LOG_LEVEL) declare -g "$key=$value" ;;
        *) warn "ignoring unknown key: $key" ;;
    esac
done < "$config_file"

The case acts as an allow-list, so an unexpected key cannot set an arbitrary variable.

Precedence, highest first:

timeout=${1:-${MYAPP_TIMEOUT:-${config_timeout:-30}}}
#         │      │                │                  └── built-in default
#         │      │                └── config file
#         │      └── environment variable
#         └── command-line flag

Nested :- expresses the whole precedence chain in one line, and matches what users expect from any well-behaved tool.

Topic 6: Knowing When to Stop

The Google Shell Style Guide puts a number on it: if a script exceeds roughly 100 lines, or does non-trivial data manipulation, rewrite it in Python.

That threshold is not arbitrary. Past it you start needing things bash does not have:

You need	Bash gives you	Cost
Nested data structures	Nothing	Parallel arrays, or string encoding
Real error types	Integer exit codes	Everything is `1`
Parsing JSON/CSV/XML	Nothing	`jq` for JSON; regex-and-hope for the rest
Floating point	Nothing	Fork to `bc` or `awk`
Unit testing	External (`bats`)	Workable, but not native
Refactoring safety	No types, no compiler	Only tests

Shell remains the right tool when the script is mostly calling other programs and the data flow is linear. It stops being the right tool the moment the logic is more interesting than the commands.

The hybrid that works well:

Keep the shell wrapper for orchestration and environment, and hand the actual computation to Python:

#!/usr/bin/env bash
set -Eeuo pipefail
SCRIPT_DIR=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)

# Shell: environment, plumbing, invoking other tools
kubectl get pods -o json |
    python3 "${SCRIPT_DIR}/../lib/analyse_pods.py" |
    while IFS=$'\t' read -r pod reason; do
        kubectl delete pod "$pod"
    done

This is a defensible architecture, not a failure. Shell is exceptional glue and a poor programming language; using it as glue plays to its strength.

Try it yourself: Count the lines in your longest script with wc -l. If it is over 100, identify the single function doing the most data manipulation — that is the one to port first.

Common mistake: Sourcing a library with a relative path (source lib/common.sh). It works from the project root and fails everywhere else, including under cron. Always resolve against ${BASH_SOURCE[0]}.

Topic 1: source Is Not import