Remote Operations & Working with APIs

Most production shell scripts do two things: run commands on other machines, and talk to HTTP APIs. Both have a small set of traps that turn a working script into a mysterious one.

Topic 1: SSH in Scripts

ssh -n -o BatchMode=yes -o ConnectTimeout=5 "$host" 'uptime'

Four options that should be on every scripted SSH call:

The stdin trap:

while read -r host; do
    ssh "$host" 'uptime'        # processes ONE host, then the loop ends
done < hosts.txt

ssh inherits the loop’s stdin and drains hosts.txt on the first iteration. Two fixes, both from the file-reading lesson:

while read -r host; do ssh -n "$host" 'uptime'; done < hosts.txt   # -n
while read -r -u 3 host; do ssh "$host" 'uptime'; done 3< hosts.txt  # own fd

Quoting for the remote side:

The command crosses a shell boundary, so it is expanded twice — once locally, once remotely.

dir=/var/log
ssh host "ls $dir"          # $dir expands LOCALLY  → ls /var/log
ssh host 'ls $dir'          # $dir expands REMOTELY → probably empty
ssh host "ls ${dir@Q}"      # bash 4.4+: shell-quote the value safely

For anything non-trivial, send a script over stdin instead of building a command string:

ssh "$host" 'bash -s' <<'REMOTE'
set -Eeuo pipefail
systemctl is-active nginx || systemctl restart nginx
REMOTE

The quoted 'REMOTE' delimiter keeps the whole block literal, so nothing expands locally. This is the cleanest way to run multi-line logic remotely.

Parallel fleets:

# Bounded parallelism with xargs
xargs -a hosts.txt -P8 -I{} ssh -n -o BatchMode=yes {} 'uptime'

# With per-host labelling
run_on() {
    local host=$1 out
    if out=$(ssh -n -o BatchMode=yes -o ConnectTimeout=5 "$host" 'uptime' 2>&1); then
        printf '%-12s OK  %s\n' "$host" "$out"
    else
        printf '%-12s ERR %s\n' "$host" "$out" >&2
        return 1
    fi
}
export -f run_on
xargs -a hosts.txt -P8 -I{} bash -c 'run_on "$@"' _ {}

For anything beyond ad-hoc work, this is where Ansible starts being the right answer — it handles inventory, idempotence, and failure reporting that you would otherwise rebuild badly.

Connection reuse:

# ~/.ssh/config
Host *
    ControlMaster auto
    ControlPath ~/.ssh/cm-%r@%h:%p
    ControlPersist 10m
    ServerAliveInterval 30

The first connection to a host opens a master; subsequent ones reuse it and skip the TCP and TLS handshake entirely. On a loop over fifty hosts this is often a 10× speedup for free.

Topic 2: Moving Files

scp file.txt host:/tmp/                       # simple, no resume, no delta
rsync -az --delete ./dist/ host:/srv/app/     # the real tool

The trailing slash rule — the most consequential detail in rsync:

rsync -a src/ dest/      # copies the CONTENTS of src into dest
rsync -a src  dest/      # copies the DIRECTORY src, creating dest/src/

One character decides between dest/index.html and dest/src/index.html. Combined with --delete, getting it wrong deletes the destination’s contents. Always --dry-run first when --delete is involved.

rsync -az --delete --dry-run ./dist/ host:/srv/app/ | head -50

Topic 3: curl as an API Client

The default curl invocation is wrong for scripts in three ways: it does not fail on HTTP errors, it prints a progress meter to stderr, and it does not time out.

curl --fail --silent --show-error --location --max-time 30 "$url"
curl -fsSL --max-time 30 "$url"        # the same thing, short form

Separating status from body:

--fail collapses every 4xx and 5xx into one exit code, which is often not enough. To branch on the actual status:

response=$(curl -sS -w '\n%{http_code}' --max-time 30 "$url")
status=${response##*$'\n'}          # last line
body=${response%$'\n'*}             # everything before it

case $status in
    200) echo "$body" | jq -r '.result' ;;
    404) echo "not found" >&2; exit 4 ;;
    429) echo "rate limited" >&2; exit 5 ;;
    5*)  echo "server error ${status}" >&2; exit 6 ;;
    *)   echo "unexpected ${status}" >&2; exit 1 ;;
esac

Writing the status on its own trailing line and splitting with parameter expansion avoids a temp file and a second request.

POSTing safely:

# Build the payload with jq so quoting and escaping are handled
payload=$(jq -n --arg name "$name" --arg env "$env" \
    '{name: $name, environment: $env, replicas: 3}')

curl -fsS --max-time 30 \
    -X POST \
    -H 'Content-Type: application/json' \
    -H "Authorization: Bearer ${API_TOKEN:?token required}" \
    -d "$payload" \
    "https://api.example.com/v1/deployments"

jq -n --arg is the correct way to build JSON. Interpolating a shell variable into a JSON string by hand breaks the first time a value contains a quote, a newline, or a backslash.

Keeping secrets out of the process list:

# WRONG -- visible to every user via `ps aux`
curl -H "Authorization: Bearer ${TOKEN}" "$url"

# Better -- a file only you can read
curl -H @auth-header.txt "$url"

# Or via netrc
curl --netrc-file ~/.netrc "$url"

Command-line arguments are world-readable on Linux. Any token passed as an argument is exposed for the lifetime of the process.

Topic 4: jq

jq '.name'                      # one field
jq -r '.name'                   # RAW — no surrounding quotes. Almost always want this
jq '.items[]'                   # iterate an array
jq '.items[] | .name'           # a field from each element
jq -r '.items[] | [.name, .status] | @tsv'    # tab-separated, for read
jq '.items | length'
jq '.items[] | select(.status == "failed")'
jq -r '.items[] | select(.cpu > 80) | .host'
jq '[.items[] | .bytes] | add'                # sum
jq -e '.items | length > 0'                   # -e: exit non-zero if false/null

-r and -e are the two flags that make jq script-friendly: raw output so values are usable directly, and a meaningful exit status so if jq -e ... works.

The handoff to shell:

jq -r '.pods[] | [.name, .phase] | @tsv' <<< "$response" |
while IFS=$'\t' read -r name phase; do
    [[ $phase == Running ]] || echo "not running: ${name}" >&2
done

@tsv handles escaping, and splitting on tab is safe because jq escapes any literal tabs in the data.

Building JSON, never by hand:

jq -n \
  --arg host "$(hostname)" \
  --arg status "$status" \
  --argjson count "$count" \
  '{host: $host, status: $status, count: $count, ts: now|todate}'

--arg passes a string, --argjson passes raw JSON (numbers, booleans, objects). This is injection-proof; string concatenation is not.

Topic 5: Retries and Idempotence

Anything crossing a network fails intermittently. A script without retry logic will page you for problems that resolve themselves.

retry() {
    local -r attempts=$1 delay=$2; shift 2
    local n=1
    until "$@"; do
        if (( n >= attempts )); then
            echo "failed after ${attempts} attempts: $*" >&2
            return 1
        fi
        echo "attempt ${n}/${attempts} failed; retrying in ${delay}s" >&2
        sleep "$delay"
        delay=$(( delay * 2 ))          # exponential backoff
        (( n++ )) || true
    done
}

retry 5 2 curl -fsS --max-time 10 "https://api.example.com/health"
retry 3 5 ssh -n -o BatchMode=yes "$host" 'systemctl is-active nginx'

Two rules about what you may retry:

• Only retry idempotent operations. A GET or a PUT is safe; retrying a POST that creates a resource may create three. Use an idempotency key if the API supports one.
• Back off exponentially. Retrying immediately, five times, against a struggling service is a denial-of-service attack you have written yourself.

Try it yourself: Write a health-check script that curls an endpoint with --max-time, retries with backoff, and exits 0/1 based on a jq assertion on the body. Run it against a URL that does not exist and confirm it fails cleanly rather than hanging.

Common mistake: Using plain curl "$url" in a cron job. Without --fail it exits 0 on a 500, without --max-time it can hang until the next run starts, and without -sS it fills the cron mail with progress bars.