fix: surface store sync failures + add secrets sync (EGB-1230, EGB-1231)

EGB-1230: `cmd_pull` synced the store with `git pull >/dev/null 2>&1` under
`set -euo pipefail`. A store that couldn't fast-forward killed the script at
that line with git's exit 128 and nothing on stdout or stderr — a banner, no
restored files, no reason, and invisible in a pipeline. The sync now routes
through `_store_sync_pull`, which guards the pull, captures git's output as
the diagnosis, and dies naming the store path and `secrets sync`. It is now
`--ff-only` to match the push path, so a plain pull can no longer manufacture
a merge commit in the store.

EGB-1231: once a store diverged there was no way out — push demanded a
fast-forward and pointed at pull, which couldn't fast-forward either, so the
advice looped and recovery meant hand-running git next to encrypted blobs.
Adds `secrets sync`: fetch, stash, rebase onto the remote, restore the stash,
then a confirmation-gated push of local commits (`--yes` to skip the prompt,
`--dry-run` to report only). Non-destructive by construction — no merge, no
force-push, no `reset --hard`, no `stash drop`; a rebase conflict names the
conflicting files and leaves the store exactly as found. `secrets which` gains
a `remote:` line reporting ahead/behind/dirty, and push's dead-end message now
points at `sync`.

test/sync.bats: 25 new tests. Full suite 353/353 green.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BrUoYuUMoTj91rzV4vxGPB
This commit is contained in:
Brian Majewski 2026-09-08 14:20:20 -07:00
parent 2558ea3c23
commit dd6a025fa0
6 changed files with 681 additions and 8 deletions

View file

@ -56,13 +56,14 @@ skips security specialist + red team, and Step 11 skips adversarial review.
## Architecture
Single bash script (`secrets`) with subcommands: init, push, pull, list, rm, rekey, verify, migrate, recipients, reencrypt, upgrade.
Single bash script (`secrets`) with subcommands: init, push, pull, list, rm, rekey, verify, migrate, recipients, reencrypt, sync, upgrade.
- Encryption: `age` with key files (not passphrases — age passphrases are non-scriptable)
- Storage: Private git repo at `~/.secrets/`
- Convention: Tracks `.env`, `.env.*`, and `.dev.vars` (not `.envrc`, `.environment-*`)
- Manifest (EGB-677 stage 1): committed `.secrets.json` is the source of truth for what syncs — `dotenv[]` (project-relative, nested ok, `@` allowed; rail rejects `..`/absolute/symlink) + `external[]` (`properties`/`file`). Push discovery auto-adds (gated by committed `options.autoAdd`, default ON; `--frozen`/`--dry-run` overrides), bootstraps the manifest on first push (written only after ≥1 blob encrypts), and absorbs a legacy `.secrets-files` (gradle-properties → `properties`; on pull the legacy file is superseded with a warning). Store layout: nested dotenv entries land at `<project>/<relpath>.age` (relpath preserved — the store self-describes where a file restores). jq is a hard dep only when a manifest exists/is written; manifest-less projects run jq-free (manifest features skipped with a notice). `check_cmd` prints platform-aware install hints.
- Store format (EGB-677 stage 2 / EGB-703): the store is self-describing via a committed one-line `$SECRETS_DIR/.secrets-format` file (`2`). Absence ⇒ v1 (every store predating EGB-703). v2's only on-disk change vs v1 is the external `properties` blob suffix: `.gradle-properties.age``.properties.age` (matching the manifest `type`); dotenv and `file` blobs are unchanged. `_store_format()` reads the marker. **Additive v2 (EGB-712):** reads resolve a `properties` blob by trying `.properties.age` then falling back to `.gradle-properties.age` (`_resolve_external_blob_read`); writes dual-write a `properties` external only when a v1 twin already exists in the store (`_external_blob_write_targets`), so existing externals keep old clients fresh while brand-new externals are written v2-only (a gentle forcing function). Blob location no longer depends on the marker — the old `_external_blob_suffix` is gone. `init` stamps a fresh store v2 (born-v2). `secrets which` prints the store-format line `format: vN`, and (EGB-700) when a `.secrets.json` is present the manifest header line also carries its schema version (`manifest (.secrets.json at <path>, version N):`). **Migration is copy-forward and non-destructive:** `secrets migrate --dry-run` (per project, reports old→new, writes nothing) → `secrets migrate` (per project, manifest-free: enumerates the store's `*.gradle-properties.age` blobs directly — same source of truth as `--finalize` — and writes their `.properties.age` twins, so a legacy `.secrets-files`-only project with no `.secrets.json` migrates cleanly and no store blob is left un-twinned; idempotent; EGB-710) → `secrets migrate --finalize` (store-wide; the ONLY destructive step — gates on `verify --all` green + every v1 blob having a v2 twin, cuts a `pre-v2-migrate-<sha>` recovery tag, stamps the marker, then drops v1 blobs; refuses without `--yes`/operator confirmation since a lagging v1 client against a finalized store stops seeing `properties` externals until it upgrades). `secrets migrate --status` is a read-only survey that walks every project in the store and reports each one's v2 readiness (v2-ready / migrated / NEEDS MIGRATE, plus a `v2-only` count of externals old clients can't read), exiting non-zero while any v1 blob is un-twinned so it gates the path to `--finalize` (EGB-710/EGB-712). **Under additive v2 (EGB-712) `--finalize` is now OPTIONAL GC, not a required milestone:** because upgraded clients dual-write existing externals and read-fall-back, *not* finalizing never cuts anyone off — finalize only reclaims the duplicate v1 blobs and stays deferrable indefinitely (defusing the cross-machine coordination gate). dotenv and `file` blobs are identical across formats, so they always propagate to old clients; only a brand-new `properties` external is v2-only. **Version-skew nudge (EGB-713):** a committed `$SECRETS_DIR/.secrets-writer-version` records the highest client `VERSION` that has written to the store (monotonic; stamped via `_stamp_writer_version` right before each store-committing `git add -A` — push/rekey/migrate/finalize — never on read paths, so it always rides a commit and never dangles to break `pull --ff-only`). `check_initialized` calls `_check_store_version_skew`, which warns once per invocation (stderr, non-fatal, `set -e`-safe) when the store's stamp is numerically greater than `_client_version` (read from `$SCRIPT_DIR/VERSION`); `secrets which` prints the `written-by:` line. Stores with no stamp (pre-EGB-713) are silent. The deliberate flatten-to-basename naming the EGB-677 CEO plan sketched was dropped as lossy (it discards the restore relpath that makes the store self-describing) — see the EGB-703 eureka. **Upgrade verb (EGB-716):** `secrets upgrade` is the fix path paired with the EGB-713 skew *warning* — it `git -C "$SCRIPT_DIR" pull --ff-only`s the tool's own checkout (fast-forward only, never merges/rewrites local commits), reports `vOLD -> vNEW`, then best-effort re-checks `_store_writer_version` against the new on-disk version so the operator sees whether the nudge is cleared (the new code takes effect next invocation). `secrets upgrade --check` does `git fetch` + `rev-list --count HEAD..@{u}` and reports availability without pulling. Deliberately thin: no auto-update, no background polling (security tool). Directed errors for not-a-git-checkout / no-upstream / diverged / offline. `cmd_upgrade` never calls `check_initialized` (it's about the tool, not the store); the skew re-check is silent unless a store with a writer-version resolves.
- Store sync + divergence (EGB-1230/EGB-1231): the store is a git repo, so a clone can end up ahead of and behind its remote at once. **EGB-1230:** `cmd_pull`'s sync used to be `git pull >/dev/null 2>&1` under `set -euo pipefail` — a store that couldn't fast-forward killed the script there with git's exit 128 and nothing on either stream (a banner, no files, no reason; invisible in a pipeline). It now routes through `_store_sync_pull`, which guards the pull, captures git's output as the diagnosis, and dies naming the store path and `secrets sync`. That sync is **`--ff-only`**, matching the push path — a plain `git pull` could quietly manufacture a merge commit in the store, and divergence is now resolved in exactly one place. **EGB-1231:** `_store_git_state` emits `ahead\tbehind\tdirty` (from `rev-list --left-right --count @{u}...HEAD` plus `status --porcelain`) and `_format_store_state` renders it; `cmd_which` prints a `remote:` line from them — offline-safe (reports against the last fetch), silent with no remote/upstream. `cmd_sync` is the reconcile verb the CLI was missing: fetch → report state → stash (`push -u`) → `rebase @{u}` → restore stash → `ensure_store_protections` (rebased-in history may lack `.gitignore`, and a store missing the `key.txt` line would stage the private key — same reasoning as push) → **confirmation-gated** `git push` of local commits. The gate (`_sync_confirm_push`) reads `/dev/tty` and requires a tty, so it stays CLOSED in scripts/CI rather than publishing to a shared store by default; `--yes` opens it, `--dry-run` reports and returns before any mutation. Non-destructive by construction: no merge, no `--force`, no `reset --hard`, no `stash drop`. A rebase conflict collects the conflicting paths BEFORE `rebase --abort` (the abort clears them), restores the stash, and dies — store byte-identical to how it was found. `_sync_restore_stash` never drops the stash on a failed pop; it tells the operator where their only copy lives. `cmd_sync` does not `_stamp_writer_version`: it replays existing commits rather than authoring content, and the stamp is specified to ride a store-committing `git add -A`. Test suite: `test/sync.bats` (25 tests), including a grep over the `cmd_sync` body asserting the destructive git verbs never appear in it.
- Verify (EGB-698): `secrets verify` is a read-only integrity check. Default mode (current project) cross-checks `$PWD/.secrets.json` against `$SECRETS_DIR/<project>/` both ways (declared-but-missing blobs + orphaned blobs) and decrypt-tests every blob (dotenv + external) by streaming plaintext to `/dev/null` (never written to disk). `secrets verify --all` decrypt-tests every blob in every project (integrity only — the store carries no manifests, so consistency can't be checked store-wide). Both recurse the whole project tree (`find -type f`, same as rekey/list). Exits non-zero on any finding so it can gate the stage-2 `migrate --finalize` and CI. The store deliberately holds no manifest — `.secrets.json` is committed in each project's own repo and read from `$PWD`.
- External files: `.secrets-files` manifest tracks designated keys from files outside the project (e.g. `~/.gradle/gradle.properties`, merged not overwritten — EGB-531) and whole binary files (type `file`, e.g. an Android upload keystore — EGB-652); see below
- Workspaces: `--workspaces` flag reads `package.json` workspaces, requires `jq`
@ -97,6 +98,7 @@ test/
manifest.bats # EGB-677 .secrets.json manifest tests (83 tests)
migrate.bats # EGB-703 store-format-v2 migration tests (35 tests)
upgrade.bats # EGB-716 `secrets upgrade` self-update tests (8 tests)
sync.bats # EGB-1230/1231 store sync + divergence reconcile tests (25 tests)
recipients.bats # EGB-283 multi-recipient age encryption tests (34 tests)
test_helper.bash # Shared setup/teardown
README.md # User-facing documentation