Project: Rescue a Wrecked Repository

Everything in this module, applied once, under conditions that resemble the real thing: something is broken, somebody is waiting, and the commands that fix it are the ones you have not typed in six months.

Build the repository, break it deliberately, and time yourself. The timings are the deliverable — a recovery you have performed once calmly is a two-minute event when it happens for real.

Topic 1: The Six Drills

Every drill follows the same three steps: establish the damage, recover, verify. The third step is the one people skip, and it is what separates a fix from a hope.

Topic 2: Building the Scenario

Write this as a script so you can reset and repeat. Roughly 40 commits, a tag, two branches, and one planted bug:

#!/bin/bash
set -euo pipefail
rm -rf /tmp/rescue && mkdir -p /tmp/rescue && cd /tmp/rescue

# A bare repo standing in for the forge, plus two clones standing in for two people
git init --bare origin.git
git clone origin.git alice
git clone origin.git bob

cd alice
git config user.name Alice && git config user.email alice@example.test

# ~40 commits of history with a tag partway through
for i in $(seq 1 40); do
  echo "line $i" >> app.txt
  # plant the regression at commit 27
  if [ "$i" -eq 27 ]; then echo "TIMEOUT=0" >> config.env; fi
  git add -A && git commit -qm "feat: step $i"
  if [ "$i" -eq 10 ]; then git tag -a v1.0.0 -m "Release 1.0.0"; fi
done

# a committed credential, twenty commits back
git show v1.0.0 >/dev/null
echo "AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY" > secrets.env
git add secrets.env && git commit -qm "chore: add local config"
for i in $(seq 41 50); do
  echo "line $i" >> app.txt && git add -A && git commit -qm "feat: step $i"
done

git push -q origin main --tags
cd ../bob && git pull -q
echo "scenario ready in /tmp/rescue"

Add a repro.sh that exits 1 when TIMEOUT=0 is present in config.env and 0 otherwise — that is your bisect oracle.

Topic 3: Drills 1–3 — Recovery

Drill 1 — a force-push destroyed a colleague’s commits.

# Bob pushes two commits. Alice, unaware, resets and force-pushes.
# Bob's commits are gone from origin.

Establish: what was the tip before the force-push, and whose commits are missing? Recover: the objects exist in Bob’s clone and in his reflog (git reflog show origin/main). Push the correct tip back with --force-with-lease. Verify: git log --oneline origin/main contains both Alice’s and Bob’s commits, and git fsck is clean. Record: the time from “Bob notices” to “origin is correct”.

Drill 2 — a committed secret.

Establish: find every commit containing it — git log -S 'wJalrXUtnFEMI' --oneline --all. Recover, in this order: rotate first (state it explicitly in your runbook, even in the drill), then git filter-repo --invert-paths --path secrets.env on a fresh mirror clone, then force-push all refs. Verify: git log -S 'wJalrXUtnFEMI' --all returns nothing, and the other clone (Bob’s) is proven to still contain the secret until he re-clones — demonstrate that, because it is the point. Record: how long a real coordination would take on top of the mechanical work.

Drill 3 — reset --hard on real work.

Establish: git reflog — find the pre-reset commit. Recover: git branch rescue <sha> rather than resetting immediately, so you can inspect first. Verify: git log --oneline rescue shows the missing commits and git diff rescue main shows exactly what was lost. Record: the elapsed time. This one should end up under a minute, which is the point of practising it.

Topic 4: Drills 4–6 — Investigation

Drill 4 — a regression with no known cause.

Establish: ./repro.sh fails at HEAD and passes at v1.0.0. Recover: git bisect start HEAD v1.0.0 then git bisect run ./repro.sh. Verify: the named commit, checked out alone, fails; its parent passes. Record: the number of tests bisect ran, and compare with log₂ of your commit range. They should match.

Drill 5 — a merge that lost changes.

Create it deliberately: on a branch, modify a function; on main, modify the same function differently; merge and resolve the conflict by taking only one side.

Establish: the code was present before the merge and absent after. git log -S '<the lost line>' --oneline --all finds where it lived. Recover: git show --cc <merge-sha> shows only the hunks a human decided — the exact place the loss happened. Reapply the missing change as a new commit (or git revert -m 1 the merge, if it is fresh and safe). Verify: both changes are present, and a test covering each passes. Record: how long it took to find the merge, versus how long to fix it. The ratio is usually 10:1, which is the argument for --cc.

Drill 6 — a rebase that duplicated the history.

Create it: rebase a branch that Bob has already pulled, force-push, and have Bob run git pull rather than re-cloning.

Establish: git log --oneline shows every commit twice with different hashes. git log --cherry-mark --left-right main...origin/main identifies equivalent pairs. Recover: discard the duplicated local lineage — git reset --hard origin/main if there is no unique work, or git rebase --onto origin/main <old-upstream-tip> <branch> to replay only the genuinely unique commits. Verify: git log --oneline | sort | uniq -d on subjects returns nothing, and no work is missing (git log --oneline <rescue-branch>..HEAD is empty). Record: what the announcement to the team should have said before the force-push.

Topic 5: What to Produce

1. A setup script that recreates the wrecked repository from scratch, deterministically. It must be re-runnable — you will want to redo drills.

2. A runbook, one page per drill:

DRILL 3 — reset --hard destroyed local commits

Symptom      "git log" no longer shows the last two days of work
Establish    git status; git reflog | head -20
Recover      git branch rescue <sha-from-reflog>
             git log --oneline rescue        # confirm before switching
             git reset --hard rescue
Verify       git log --oneline -5 shows the recovered commits
             git diff rescue HEAD is empty
Measured     47 seconds
Prevention   commit more often; git tag backup/… before destructive commands

Write it for someone who has not read this module. Exact commands, no prose.

3. A prevention note mapping each incident to a control:

Notice that “be more careful” appears nowhere. Every row is a setting, a hook, or a habit with a command attached.

Topic 6: The Questions This Should Leave You Able to Answer

Without notes, from your own runbook:

• Where do lost commits actually live, and what is the one category Git cannot recover?
• Why does removing a secret from history not protect the secret?
• What is the difference between --force and --force-with-lease, in terms of what each one checks?
• Why does a squash merge make git branch -d report the branch as unmerged?
• Which command shows only the lines a human decided during a merge?
• What does git bisect run need from your script, including exit code 125?
• Which forge setting would have prevented drill 1 entirely, and why does “include administrators” matter?

If any of those is uncertain, the drill for it is the one to repeat — that is the whole purpose of having built a script that resets the scenario.

Common mistake: doing the drills once, successfully, and not writing down the timings. The runbook is what you will actually open at 6pm during the real version, and a runbook without measured timings gives you no way to tell whether the recovery you are attempting is on track or has gone wrong.