diff --git a/CLAUDE.md b/CLAUDE.md index bd2954f..9693fde 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -56,13 +56,13 @@ skips security specialist + red team, and Step 11 skips adversarial review. ## Architecture -Single bash script (`secrets`) with subcommands: init, push, pull, list, rm, rekey, verify, migrate. +Single bash script (`secrets`) with subcommands: init, push, pull, list, rm, rekey, verify, migrate, upgrade. - Encryption: `age` with key files (not passphrases — age passphrases are non-scriptable) - Storage: Private git repo at `~/.secrets/` - Convention: Tracks `.env`, `.env.*`, and `.dev.vars` (not `.envrc`, `.environment-*`) - Manifest (EGB-677 stage 1): committed `.secrets.json` is the source of truth for what syncs — `dotenv[]` (project-relative, nested ok, `@` allowed; rail rejects `..`/absolute/symlink) + `external[]` (`properties`/`file`). Push discovery auto-adds (gated by committed `options.autoAdd`, default ON; `--frozen`/`--dry-run` overrides), bootstraps the manifest on first push (written only after ≥1 blob encrypts), and absorbs a legacy `.secrets-files` (gradle-properties → `properties`; on pull the legacy file is superseded with a warning). Store layout: nested dotenv entries land at `/.age` (relpath preserved — the store self-describes where a file restores). jq is a hard dep only when a manifest exists/is written; manifest-less projects run jq-free (manifest features skipped with a notice). `check_cmd` prints platform-aware install hints. -- Store format (EGB-677 stage 2 / EGB-703): the store is self-describing via a committed one-line `$SECRETS_DIR/.secrets-format` file (`2`). Absence ⇒ v1 (every store predating EGB-703). v2's only on-disk change vs v1 is the external `properties` blob suffix: `.gradle-properties.age` → `.properties.age` (matching the manifest `type`); dotenv and `file` blobs are unchanged. `_store_format()` reads the marker. **Additive v2 (EGB-712):** reads resolve a `properties` blob by trying `.properties.age` then falling back to `.gradle-properties.age` (`_resolve_external_blob_read`); writes dual-write a `properties` external only when a v1 twin already exists in the store (`_external_blob_write_targets`), so existing externals keep old clients fresh while brand-new externals are written v2-only (a gentle forcing function). Blob location no longer depends on the marker — the old `_external_blob_suffix` is gone. `init` stamps a fresh store v2 (born-v2). `secrets which` prints the store-format line `format: vN`, and (EGB-700) when a `.secrets.json` is present the manifest header line also carries its schema version (`manifest (.secrets.json at , version N):`). **Migration is copy-forward and non-destructive:** `secrets migrate --dry-run` (per project, reports old→new, writes nothing) → `secrets migrate` (per project, manifest-free: enumerates the store's `*.gradle-properties.age` blobs directly — same source of truth as `--finalize` — and writes their `.properties.age` twins, so a legacy `.secrets-files`-only project with no `.secrets.json` migrates cleanly and no store blob is left un-twinned; idempotent; EGB-710) → `secrets migrate --finalize` (store-wide; the ONLY destructive step — gates on `verify --all` green + every v1 blob having a v2 twin, cuts a `pre-v2-migrate-` recovery tag, stamps the marker, then drops v1 blobs; refuses without `--yes`/operator confirmation since a lagging v1 client against a finalized store stops seeing `properties` externals until it upgrades). `secrets migrate --status` is a read-only survey that walks every project in the store and reports each one's v2 readiness (v2-ready / migrated / NEEDS MIGRATE, plus a `v2-only` count of externals old clients can't read), exiting non-zero while any v1 blob is un-twinned so it gates the path to `--finalize` (EGB-710/EGB-712). **Under additive v2 (EGB-712) `--finalize` is now OPTIONAL GC, not a required milestone:** because upgraded clients dual-write existing externals and read-fall-back, *not* finalizing never cuts anyone off — finalize only reclaims the duplicate v1 blobs and stays deferrable indefinitely (defusing the cross-machine coordination gate). dotenv and `file` blobs are identical across formats, so they always propagate to old clients; only a brand-new `properties` external is v2-only. **Version-skew nudge (EGB-713):** a committed `$SECRETS_DIR/.secrets-writer-version` records the highest client `VERSION` that has written to the store (monotonic; stamped via `_stamp_writer_version` right before each store-committing `git add -A` — push/rekey/migrate/finalize — never on read paths, so it always rides a commit and never dangles to break `pull --ff-only`). `check_initialized` calls `_check_store_version_skew`, which warns once per invocation (stderr, non-fatal, `set -e`-safe) when the store's stamp is numerically greater than `_client_version` (read from `$SCRIPT_DIR/VERSION`); `secrets which` prints the `written-by:` line. Stores with no stamp (pre-EGB-713) are silent. The deliberate flatten-to-basename naming the EGB-677 CEO plan sketched was dropped as lossy (it discards the restore relpath that makes the store self-describing) — see the EGB-703 eureka. +- Store format (EGB-677 stage 2 / EGB-703): the store is self-describing via a committed one-line `$SECRETS_DIR/.secrets-format` file (`2`). Absence ⇒ v1 (every store predating EGB-703). v2's only on-disk change vs v1 is the external `properties` blob suffix: `.gradle-properties.age` → `.properties.age` (matching the manifest `type`); dotenv and `file` blobs are unchanged. `_store_format()` reads the marker. **Additive v2 (EGB-712):** reads resolve a `properties` blob by trying `.properties.age` then falling back to `.gradle-properties.age` (`_resolve_external_blob_read`); writes dual-write a `properties` external only when a v1 twin already exists in the store (`_external_blob_write_targets`), so existing externals keep old clients fresh while brand-new externals are written v2-only (a gentle forcing function). Blob location no longer depends on the marker — the old `_external_blob_suffix` is gone. `init` stamps a fresh store v2 (born-v2). `secrets which` prints the store-format line `format: vN`, and (EGB-700) when a `.secrets.json` is present the manifest header line also carries its schema version (`manifest (.secrets.json at , version N):`). **Migration is copy-forward and non-destructive:** `secrets migrate --dry-run` (per project, reports old→new, writes nothing) → `secrets migrate` (per project, manifest-free: enumerates the store's `*.gradle-properties.age` blobs directly — same source of truth as `--finalize` — and writes their `.properties.age` twins, so a legacy `.secrets-files`-only project with no `.secrets.json` migrates cleanly and no store blob is left un-twinned; idempotent; EGB-710) → `secrets migrate --finalize` (store-wide; the ONLY destructive step — gates on `verify --all` green + every v1 blob having a v2 twin, cuts a `pre-v2-migrate-` recovery tag, stamps the marker, then drops v1 blobs; refuses without `--yes`/operator confirmation since a lagging v1 client against a finalized store stops seeing `properties` externals until it upgrades). `secrets migrate --status` is a read-only survey that walks every project in the store and reports each one's v2 readiness (v2-ready / migrated / NEEDS MIGRATE, plus a `v2-only` count of externals old clients can't read), exiting non-zero while any v1 blob is un-twinned so it gates the path to `--finalize` (EGB-710/EGB-712). **Under additive v2 (EGB-712) `--finalize` is now OPTIONAL GC, not a required milestone:** because upgraded clients dual-write existing externals and read-fall-back, *not* finalizing never cuts anyone off — finalize only reclaims the duplicate v1 blobs and stays deferrable indefinitely (defusing the cross-machine coordination gate). dotenv and `file` blobs are identical across formats, so they always propagate to old clients; only a brand-new `properties` external is v2-only. **Version-skew nudge (EGB-713):** a committed `$SECRETS_DIR/.secrets-writer-version` records the highest client `VERSION` that has written to the store (monotonic; stamped via `_stamp_writer_version` right before each store-committing `git add -A` — push/rekey/migrate/finalize — never on read paths, so it always rides a commit and never dangles to break `pull --ff-only`). `check_initialized` calls `_check_store_version_skew`, which warns once per invocation (stderr, non-fatal, `set -e`-safe) when the store's stamp is numerically greater than `_client_version` (read from `$SCRIPT_DIR/VERSION`); `secrets which` prints the `written-by:` line. Stores with no stamp (pre-EGB-713) are silent. The deliberate flatten-to-basename naming the EGB-677 CEO plan sketched was dropped as lossy (it discards the restore relpath that makes the store self-describing) — see the EGB-703 eureka. **Upgrade verb (EGB-716):** `secrets upgrade` is the fix path paired with the EGB-713 skew *warning* — it `git -C "$SCRIPT_DIR" pull --ff-only`s the tool's own checkout (fast-forward only, never merges/rewrites local commits), reports `vOLD -> vNEW`, then best-effort re-checks `_store_writer_version` against the new on-disk version so the operator sees whether the nudge is cleared (the new code takes effect next invocation). `secrets upgrade --check` does `git fetch` + `rev-list --count HEAD..@{u}` and reports availability without pulling. Deliberately thin: no auto-update, no background polling (security tool). Directed errors for not-a-git-checkout / no-upstream / diverged / offline. `cmd_upgrade` never calls `check_initialized` (it's about the tool, not the store); the skew re-check is silent unless a store with a writer-version resolves. - Verify (EGB-698): `secrets verify` is a read-only integrity check. Default mode (current project) cross-checks `$PWD/.secrets.json` against `$SECRETS_DIR//` both ways (declared-but-missing blobs + orphaned blobs) and decrypt-tests every blob (dotenv + external) by streaming plaintext to `/dev/null` (never written to disk). `secrets verify --all` decrypt-tests every blob in every project (integrity only — the store carries no manifests, so consistency can't be checked store-wide). Both recurse the whole project tree (`find -type f`, same as rekey/list). Exits non-zero on any finding so it can gate the stage-2 `migrate --finalize` and CI. The store deliberately holds no manifest — `.secrets.json` is committed in each project's own repo and read from `$PWD`. - External files: `.secrets-files` manifest tracks designated keys from files outside the project (e.g. `~/.gradle/gradle.properties`, merged not overwritten — EGB-531) and whole binary files (type `file`, e.g. an Android upload keystore — EGB-652); see below - Workspaces: `--workspaces` flag reads `package.json` workspaces, requires `jq` @@ -78,6 +78,7 @@ test/ secrets.bats # bats-core test suite (140 tests) manifest.bats # EGB-677 .secrets.json manifest tests (83 tests) migrate.bats # EGB-703 store-format-v2 migration tests (35 tests) + upgrade.bats # EGB-716 `secrets upgrade` self-update tests (8 tests) test_helper.bash # Shared setup/teardown README.md # User-facing documentation CLAUDE.md # This file diff --git a/README.md b/README.md index 4fa3459..aa4662c 100644 --- a/README.md +++ b/README.md @@ -187,6 +187,8 @@ secrets clear | `secrets migrate [--dry-run]` | Copy-forward this project's encrypted blobs to store format v2 (non-destructive; manifest-free; `--dry-run` previews) | | `secrets migrate --status` | Survey every project's v2 readiness; exits non-zero until the whole store is finalize-ready | | `secrets migrate --finalize` | **Optional GC** — drop the old v1 blobs and mark the store pure v2. Never required: upgraded clients dual-write and read-fall-back, so not finalizing never cuts anyone off | +| `secrets upgrade` | Self-update the tool: `git pull --ff-only` on the `secrets` checkout, report old → new version, then re-check store version-skew. No auto-update, no background checks | +| `secrets upgrade --check` | Report whether an update is available (without pulling); changes nothing | ### Upgrading: do teammates on an older `secrets` get new secrets? diff --git a/secrets b/secrets index 2d55aa7..f567242 100755 --- a/secrets +++ b/secrets @@ -2375,6 +2375,90 @@ cmd_which() { fi } +# `secrets upgrade [--check]` (EGB-716) — self-update the TOOL checkout. +# +# Pairs the EGB-713 skew WARNING with a fix path. Deliberately thin and explicit +# (no auto-update, no background polling — this is a security tool): it only +# fast-forwards the tool's own git checkout, never merges or rewrites local +# commits. --check reports whether an update is available and changes nothing. +# After a real update it best-effort re-checks the store's writer-version skew +# against the NEW on-disk version, so the operator sees whether the EGB-713 +# nudge is now cleared (the new code itself takes effect on the next command). +cmd_upgrade() { + local check_only=0 + while [ $# -gt 0 ]; do + case "$1" in + --check) check_only=1; shift ;; + -*) die "Unknown upgrade flag: $1. Usage: secrets upgrade [--check]" ;; + *) die "Unexpected argument to upgrade: $1. Usage: secrets upgrade [--check]" ;; + esac + done + + check_cmd git + + # Self-update only works on a git checkout of the tool. + if ! git -C "$SCRIPT_DIR" rev-parse --is-inside-work-tree >/dev/null 2>&1; then + die "secrets at $SCRIPT_DIR is not a git checkout, so it can't self-update. + Re-install by cloning the tool repo, e.g.: git clone " + fi + + # Need a tracking branch to compare against / pull from. + local upstream + upstream=$(git -C "$SCRIPT_DIR" rev-parse --abbrev-ref --symbolic-full-name '@{u}' 2>/dev/null || true) + if [ -z "$upstream" ]; then + die "No upstream tracking branch for the secrets checkout at $SCRIPT_DIR. + Set one with: git -C \"$SCRIPT_DIR\" branch --set-upstream-to=origin/main" + fi + + local oldver; oldver=$(_client_version) + + if [ "$check_only" = 1 ]; then + if ! git -C "$SCRIPT_DIR" fetch --quiet 2>/dev/null; then + die "Couldn't reach the tool remote to check for updates (offline?). + Try again when connected, or run: git -C \"$SCRIPT_DIR\" fetch" + fi + local behind; behind=$(git -C "$SCRIPT_DIR" rev-list --count "HEAD..$upstream" 2>/dev/null || echo 0) + if [ "${behind:-0}" -gt 0 ]; then + info "Update available: $behind commit(s) behind $upstream (you're on v$oldver)." + info "Apply it with: secrets upgrade" + else + info "secrets is up to date (v$oldver)." + fi + return 0 + fi + + info "Updating secrets at $SCRIPT_DIR ..." + # Fast-forward only: never merge or rewrite local commits. + if ! git -C "$SCRIPT_DIR" pull --ff-only 2>&1; then + die "Update failed (see git output above). + Likely a local change or a diverged branch in $SCRIPT_DIR. + Inspect with: git -C \"$SCRIPT_DIR\" status" + fi + + local newver; newver=$(_client_version) + if [ "$oldver" = "$newver" ]; then + info "Already up to date (v$newver)." + else + info "Upgraded: v$oldver -> v$newver" + info "The new version takes effect on your next 'secrets' command." + fi + + # Best-effort EGB-713 skew re-check against the NEW version. Silent unless a + # store with a recorded writer-version resolves. + resolve_store 2>/dev/null || true + if [ -n "${SECRETS_DIR:-}" ] && [ -f "$SECRETS_DIR/$WRITER_VERSION_FILE_NAME" ]; then + local sv; sv=$(_store_writer_version) + if [ -n "$sv" ]; then + if _version_gt "$sv" "$newver"; then + info "Note: the store was last written by v$sv — still ahead of v$newver. Another machine may run a newer client." + else + info "Your client (v$newver) is now at or ahead of the store's last writer (v$sv)." + fi + fi + fi + return 0 +} + # Decrypt-test one blob with the current key. Plaintext is streamed to # /dev/null and never written to disk (read-only contract). Returns 0 if the # blob decrypts, non-zero otherwise. @@ -2783,6 +2867,8 @@ Usage: secrets which Show the active store, manifest, and external entries secrets where Alias for `which` secrets status Alias for `which` + secrets upgrade Self-update the tool (git pull --ff-only) + recheck skew + secrets upgrade --check Report whether an update is available; change nothing Tracked files: .env, .env.*, .dev.vars @@ -2952,6 +3038,7 @@ case "${1:-help}" in verify) shift; cmd_verify "$@" ;; migrate) shift; cmd_migrate "$@" ;; which|where|status) cmd_which ;; + upgrade) shift; cmd_upgrade "$@" ;; help|--help|-h) cmd_help ;; *) die "Unknown command: $1. Run 'secrets help' for usage." ;; esac diff --git a/test/upgrade.bats b/test/upgrade.bats new file mode 100644 index 0000000..041bacb --- /dev/null +++ b/test/upgrade.bats @@ -0,0 +1,115 @@ +#!/usr/bin/env bats +# EGB-716: `secrets upgrade` verb — self-update (git pull --ff-only) + skew re-check. +# +# These tests never touch the real tool checkout. Each test relocates a COPY of +# the script into a throwaway git repo wired to a bare upstream, so $SCRIPT_DIR +# (computed from BASH_SOURCE) resolves to the fake tool repo and the pull/fetch +# operate there. + +load test_helper + +# Create a fake tool repo at $TOOL (script copy + VERSION), wired to a bare +# upstream at $TOOL_REMOTE, at version $1. cd's into $TOOL (under $HOME so +# resolve_store's walk-up stays bounded and never strays to a real store). +setup_tool_repo() { + TOOL="$TEST_TMPDIR/tool" + TOOL_REMOTE="$TEST_TMPDIR/tool-remote.git" + mkdir -p "$TOOL" + cp "$SECRETS_BIN" "$TOOL/secrets" + echo "$1" > "$TOOL/VERSION" + git -c init.defaultBranch=main init -q "$TOOL" + git -C "$TOOL" add -A + git -C "$TOOL" -c user.email=t@t -c user.name=t commit -qm "v$1" + git -c init.defaultBranch=main init --bare -q "$TOOL_REMOTE" + git -C "$TOOL" remote add origin "$TOOL_REMOTE" + git -C "$TOOL" push -q -u origin HEAD:main + cd "$TOOL" +} + +# Publish a newer VERSION to the upstream (as a different clone would). +advance_tool_remote() { + local clone="$TEST_TMPDIR/tool-pub" + rm -rf "$clone" + git clone -q "$TOOL_REMOTE" "$clone" + echo "$1" > "$clone/VERSION" + git -C "$clone" -c user.email=t@t -c user.name=t commit -qam "v$1" + git -C "$clone" push -q origin HEAD:main + rm -rf "$clone" +} + +@test "upgrade --check reports an available update without changing VERSION (EGB-716)" { + setup_tool_repo 0.1.0.0 + advance_tool_remote 0.2.0.0 + run "$TOOL/secrets" upgrade --check + [ "$status" -eq 0 ] + [[ "$output" == *"Update available"* ]] || false + [[ "$output" == *"0.1.0.0"* ]] || false + # --check must not pull: local VERSION is untouched. + [ "$(cat "$TOOL/VERSION")" = "0.1.0.0" ] +} + +@test "upgrade --check is clean when already current (EGB-716)" { + setup_tool_repo 0.2.0.0 + run "$TOOL/secrets" upgrade --check + [ "$status" -eq 0 ] + [[ "$output" == *"up to date"* ]] || false +} + +@test "upgrade fast-forwards and reports old -> new (EGB-716)" { + setup_tool_repo 0.1.0.0 + advance_tool_remote 0.2.0.0 + run "$TOOL/secrets" upgrade + [ "$status" -eq 0 ] + [[ "$output" == *"v0.1.0.0 -> v0.2.0.0"* ]] || false + [ "$(cat "$TOOL/VERSION")" = "0.2.0.0" ] +} + +@test "upgrade is a no-op when already at the latest (EGB-716)" { + setup_tool_repo 0.2.0.0 + run "$TOOL/secrets" upgrade + [ "$status" -eq 0 ] + [[ "$output" == *"up to date"* ]] || false + [ "$(cat "$TOOL/VERSION")" = "0.2.0.0" ] +} + +@test "upgrade refuses when the tool dir is not a git checkout (EGB-716)" { + local d="$HOME/plain-tool" + mkdir -p "$d" + cp "$SECRETS_BIN" "$d/secrets" + echo 0.1.0.0 > "$d/VERSION" + cd "$d" + run "$d/secrets" upgrade + [ "$status" -eq 1 ] + [[ "$output" == *"git checkout"* ]] || false +} + +@test "upgrade rejects an unknown flag (EGB-716)" { + setup_tool_repo 0.1.0.0 + run "$TOOL/secrets" upgrade --bogus + [ "$status" -eq 1 ] + [[ "$output" == *"Unknown upgrade flag"* ]] || false +} + +@test "upgrade re-checks store skew and confirms the client caught up (EGB-716)" { + setup_tool_repo 0.1.0.0 + advance_tool_remote 0.9.0.0 + # A store last written by a newer client than our starting version. + git -c init.defaultBranch=main init -q "$SECRETS_DIR" + echo 0.8.0.0 > "$SECRETS_DIR/.secrets-writer-version" + run "$TOOL/secrets" upgrade + [ "$status" -eq 0 ] + [[ "$output" == *"v0.1.0.0 -> v0.9.0.0"* ]] || false + # New client (0.9.0.0) is now ahead of the store's last writer (0.8.0.0). + [[ "$output" == *"at or ahead"* ]] || false +} + +@test "upgrade still notes when the store is ahead of the upgraded client (EGB-716)" { + setup_tool_repo 0.1.0.0 + advance_tool_remote 0.2.0.0 + git -c init.defaultBranch=main init -q "$SECRETS_DIR" + echo 0.9.0.0 > "$SECRETS_DIR/.secrets-writer-version" + run "$TOOL/secrets" upgrade + [ "$status" -eq 0 ] + [[ "$output" == *"v0.1.0.0 -> v0.2.0.0"* ]] || false + [[ "$output" == *"still ahead"* ]] || false +}