From e2ad661da5e306a4a1e882a211b26fd43423d00e Mon Sep 17 00:00:00 2001 From: Brian Majewski Date: Sun, 7 Jun 2026 16:07:31 -0700 Subject: [PATCH] feat: store-format-v2 self-describing migration (EGB-703, folds in EGB-700) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Stage 2 of EGB-677. Makes the store self-describing and unifies the legacy external `properties` blob suffix, via a copy-forward migration that never destroys data until an explicit, gated finalize. Scope decision (see eureka): the EGB-677 CEO plan's "flatten dotenv blobs to basename" was dropped as LOSSY — it discards the restore relpath that makes the store self-describing and adds basename collisions. Engineering analysis (4 parallel design agents) showed the store is already relpath-self-describing; the only real v1→v2 delta is the `properties` suffix. This implements the minimal, safe v2 that achieves the epic's self-describing goal. What's added: - `.secrets-format` marker (committed, one line `2`). Absence ⇒ v1 (every pre-EGB-703 store). `_store_format()` reads it; `init` stamps fresh stores born-v2. `secrets which` prints `format: vN` (EGB-700 folded in). - `_external_blob_suffix(type)` — single source of truth for the external suffix (v2: gradle-properties → properties; file unchanged). push/pull/verify all route through it, so v1 and v2 stores never disagree on blob location. - `secrets migrate` — per-project copy-forward (writes `.properties.age` twins beside v1 blobs; idempotent; needs the project manifest), `--dry-run` (reports old→new, writes nothing), `--finalize` (store-wide, the only destructive step: gates on `verify --all` green + every v1 blob twinned, cuts a `pre-v2-migrate-` recovery tag, stamps the marker, then drops v1 blobs; refuses without `--yes`/operator confirmation). rekey and verify --all stay format-agnostic (recursive find walk) — no change. 21 new bats tests (test/migrate.bats): marker/born-v2, format-aware suffix, v1 back-compat, dry-run, copy-forward idempotency, no-manifest die, finalize gates (verify-not-green refusal, untwinned refusal, recovery tag, confirmation), and full v1→window→finalize round-trip. Updated 4 existing tests for the born-v2 suffix. Full suite 231/231, bash 3.2 clean. --- CLAUDE.md | 5 +- README.md | 2 + secrets | 212 ++++++++++++++++++++++++++++++++++- test/manifest.bats | 4 +- test/migrate.bats | 270 +++++++++++++++++++++++++++++++++++++++++++++ test/secrets.bats | 4 +- 6 files changed, 487 insertions(+), 10 deletions(-) create mode 100644 test/migrate.bats diff --git a/CLAUDE.md b/CLAUDE.md index b6c380f..e953300 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -56,12 +56,13 @@ skips security specialist + red team, and Step 11 skips adversarial review. ## Architecture -Single bash script (`secrets`) with subcommands: init, push, pull, list, rm, rekey, verify. +Single bash script (`secrets`) with subcommands: init, push, pull, list, rm, rekey, verify, migrate. - Encryption: `age` with key files (not passphrases — age passphrases are non-scriptable) - Storage: Private git repo at `~/.secrets/` - Convention: Tracks `.env`, `.env.*`, and `.dev.vars` (not `.envrc`, `.environment-*`) -- Manifest (EGB-677 stage 1): committed `.secrets.json` is the source of truth for what syncs — `dotenv[]` (project-relative, nested ok, `@` allowed; rail rejects `..`/absolute/symlink) + `external[]` (`properties`/`file`). Push discovery auto-adds (gated by committed `options.autoAdd`, default ON; `--frozen`/`--dry-run` overrides), bootstraps the manifest on first push (written only after ≥1 blob encrypts), and absorbs a legacy `.secrets-files` (gradle-properties → `properties`; on pull the legacy file is superseded with a warning). v1 store layout unchanged in stage 1: nested entries land at `/.age`; `properties` blobs keep the legacy `.gradle-properties.age` suffix until the stage-2 store migration. jq is a hard dep only when a manifest exists/is written; manifest-less projects run jq-free (manifest features skipped with a notice). `check_cmd` prints platform-aware install hints. +- Manifest (EGB-677 stage 1): committed `.secrets.json` is the source of truth for what syncs — `dotenv[]` (project-relative, nested ok, `@` allowed; rail rejects `..`/absolute/symlink) + `external[]` (`properties`/`file`). Push discovery auto-adds (gated by committed `options.autoAdd`, default ON; `--frozen`/`--dry-run` overrides), bootstraps the manifest on first push (written only after ≥1 blob encrypts), and absorbs a legacy `.secrets-files` (gradle-properties → `properties`; on pull the legacy file is superseded with a warning). Store layout: nested dotenv entries land at `/.age` (relpath preserved — the store self-describes where a file restores). jq is a hard dep only when a manifest exists/is written; manifest-less projects run jq-free (manifest features skipped with a notice). `check_cmd` prints platform-aware install hints. +- Store format (EGB-677 stage 2 / EGB-703): the store is self-describing via a committed one-line `$SECRETS_DIR/.secrets-format` file (`2`). Absence ⇒ v1 (every store predating EGB-703). v2's only on-disk change vs v1 is the external `properties` blob suffix: `.gradle-properties.age` → `.properties.age` (matching the manifest `type`); dotenv and `file` blobs are unchanged. `_store_format()` reads the marker; `_external_blob_suffix(type)` is the single source of truth for the suffix (push/pull/verify all route through it, so v1 and v2 stores never disagree on where a blob lives). `init` stamps a fresh store v2 (born-v2). `secrets which` prints `format: vN`. **Migration is copy-forward and non-destructive:** `secrets migrate --dry-run` (per project, reports old→new, writes nothing) → `secrets migrate` (per project, writes `.properties.age` twins beside the v1 blobs; needs the project manifest to know which externals are `properties`; idempotent) → `secrets migrate --finalize` (store-wide; the ONLY destructive step — gates on `verify --all` green + every v1 blob having a v2 twin, cuts a `pre-v2-migrate-` recovery tag, stamps the marker, then drops v1 blobs; refuses without `--yes`/operator confirmation since a lagging v1 client against a finalized store stops seeing `properties` externals until it upgrades). The deliberate flatten-to-basename naming the EGB-677 CEO plan sketched was dropped as lossy (it discards the restore relpath that makes the store self-describing) — see the EGB-703 eureka. - Verify (EGB-698): `secrets verify` is a read-only integrity check. Default mode (current project) cross-checks `$PWD/.secrets.json` against `$SECRETS_DIR//` both ways (declared-but-missing blobs + orphaned blobs) and decrypt-tests every blob (dotenv + external) by streaming plaintext to `/dev/null` (never written to disk). `secrets verify --all` decrypt-tests every blob in every project (integrity only — the store carries no manifests, so consistency can't be checked store-wide). Both recurse the whole project tree (`find -type f`, same as rekey/list). Exits non-zero on any finding so it can gate the stage-2 `migrate --finalize` and CI. The store deliberately holds no manifest — `.secrets.json` is committed in each project's own repo and read from `$PWD`. - External files: `.secrets-files` manifest tracks designated keys from files outside the project (e.g. `~/.gradle/gradle.properties`, merged not overwritten — EGB-531) and whole binary files (type `file`, e.g. an Android upload keystore — EGB-652); see below - Workspaces: `--workspaces` flag reads `package.json` workspaces, requires `jq` diff --git a/README.md b/README.md index b92f7a5..354e23c 100644 --- a/README.md +++ b/README.md @@ -171,6 +171,8 @@ secrets clear | `secrets rekey` | Generate a new encryption key and re-encrypt everything | | `secrets verify [project]` | Check the current project's `.secrets.json` against the store (missing/orphaned blobs) and decrypt every blob. `[project]` overrides the store directory name; the manifest is still read from the current directory | | `secrets verify --all` | Decrypt-test every blob in every project — a store-wide integrity sweep | +| `secrets migrate [--dry-run]` | Copy-forward this project's encrypted blobs to store format v2 (non-destructive; `--dry-run` previews) | +| `secrets migrate --finalize` | Drop the old v1 blobs and mark the store v2 — runs once, store-wide, after `verify` is green and every machine is upgraded | ### Automatic project detection diff --git a/secrets b/secrets index 5ed68a5..600a25e 100755 --- a/secrets +++ b/secrets @@ -517,6 +517,39 @@ _secrets_files_slug() { printf '%s-%s' "$clean" "$sum" } +# ─── Store format (EGB-703) ─────────────────────────────────────────── +# +# The store self-describes its format via a committed one-line file +# `$SECRETS_DIR/.secrets-format` containing `2`. Absence (or any non-`2` +# content) means format v1 — the legacy default for every store that +# predates EGB-703. `init` stamps a fresh store v2 (born-v2); `migrate +# --finalize` stamps a migrated store v2. Requires resolve_store to have +# run (SECRETS_DIR set). +STORE_FORMAT_FILE_NAME=".secrets-format" +_store_format() { + local f="$SECRETS_DIR/$STORE_FORMAT_FILE_NAME" v + if [ -f "$f" ]; then + v=$(head -1 "$f" 2>/dev/null | tr -dc '0-9') + [ "$v" = "2" ] && { echo 2; return; } + fi + echo 1 +} + +# The on-disk blob suffix for an external entry, format-aware. v2 unifies +# the legacy `gradle-properties` suffix to `properties` (matching the JSON +# manifest `type`); `file` is unchanged in both formats. The slug + this +# suffix + `.age` is the external blob name. This is the single source of +# truth for the suffix — push, pull, verify all route through it so a v1 +# and a v2 store can never disagree on where a blob lives. +_external_blob_suffix() { + local mtype="$1" + if [ "$mtype" = "gradle-properties" ] && [ "$(_store_format)" = "2" ]; then + echo "properties" + else + echo "$mtype" + fi +} + # Merge managed key=value lines (from $2) into target file $1, preserving # all unrelated lines/comments/order. Updates a managed key in place (first # occurrence), collapses duplicates, appends new keys. Atomic + mode-safe. @@ -615,7 +648,7 @@ push_external_files() { # EGB-652: whole-file sync — encrypt the file verbatim (binary-safe). mkdir -p "$SECRETS_DIR/$project/external" local fslug; fslug=$(_secrets_files_slug "$mpath") - age -r "$pubkey" -o "$SECRETS_DIR/$project/external/$fslug.file.age" "$expanded" + age -r "$pubkey" -o "$SECRETS_DIR/$project/external/$fslug.$(_external_blob_suffix file).age" "$expanded" info "Encrypted file $mpath" pushed=$((pushed + 1)) continue @@ -644,7 +677,7 @@ push_external_files() { fi mkdir -p "$SECRETS_DIR/$project/external" local slug; slug=$(_secrets_files_slug "$mpath") - age -r "$pubkey" -o "$SECRETS_DIR/$project/external/$slug.$mtype.age" "$tmp" + age -r "$pubkey" -o "$SECRETS_DIR/$project/external/$slug.$(_external_blob_suffix "$mtype").age" "$tmp" rm -f "$tmp" info "Extracted $found key(s) from $mpath" pushed=$((pushed + 1)) @@ -673,7 +706,7 @@ pull_external_files() { continue fi local slug; slug=$(_secrets_files_slug "$mpath") - local blob="$SECRETS_DIR/$project/external/$slug.$mtype.age" + local blob="$SECRETS_DIR/$project/external/$slug.$(_external_blob_suffix "$mtype").age" if [ ! -f "$blob" ]; then echo "WARNING: $SECRETS_FILES_NAME names '$mpath' but no encrypted data exists in the store yet. Run 'secrets push' on a machine that has these keys. Skipping." >&2 continue @@ -1126,6 +1159,11 @@ Your key file has been left untouched." # Write .gitignore write_store_gitignore + # Stamp the store format (EGB-703): a fresh store is born v2 — it has no + # v1 blobs, so it is already in v2 shape. The marker is a committed, + # non-secret metadata file (NOT gitignored); the first push stages it. + printf '2\n' > "$SECRETS_DIR/$STORE_FORMAT_FILE_NAME" + # Install pre-commit hook mkdir -p "$SECRETS_DIR/.git/hooks" install_hook @@ -1866,6 +1904,9 @@ cmd_which() { resolve_store echo "store: $SECRETS_DIR" echo "source: $STORE_SOURCE" + # EGB-700: surface the store format so users can tell v1 from v2 during the + # migration window. A v1 store is a legacy store with no format marker. + echo "format: v$(_store_format)" # v2 manifest (.secrets.json): validate and summarize. Validation here # is deliberately fatal (symlink / malformed / unsupported version) so @@ -2005,7 +2046,7 @@ _verify_project() { while IFS=$'\t' read -r etype epath _; do [ -n "$etype" ] || continue slug=$(_secrets_files_slug "$epath") - erel="external/$slug.$etype.age" + erel="external/$slug.$(_external_blob_suffix "$etype").age" expected="$expected$erel"$'\n' eblob="$pdir/$erel" if [ ! -f "$eblob" ]; then @@ -2066,6 +2107,166 @@ cmd_verify() { fi } +# ─── Store-format-v2 migration (EGB-703) ────────────────────────────── +# +# v2 renames the legacy `properties` blob suffix (.gradle-properties.age → +# .properties.age) and marks the store self-describing via .secrets-format. +# Migration is copy-forward and non-destructive until --finalize: +# secrets migrate --dry-run # per project: report old→new, write nothing +# secrets migrate # per project: write v2 twins beside v1 blobs +# secrets migrate --finalize # store-wide: verify, drop v1, stamp v2 +# Per-project (needs the project manifest to know which externals are +# `properties`); finalize is store-wide. Mirrors verify's project/--all split. + +# Copy-forward (or dry-run preview) for the current project. Reads +# $PWD/.secrets.json; only `properties` external blobs rename in v2. +_migrate_project() { + local dry_run="$1" + if [ "$(_store_format)" = "2" ]; then + info "Store is already format v2 — nothing to migrate." + return 0 + fi + local manifest="$PWD/$SECRETS_JSON_NAME" + if [ ! -e "$manifest" ]; then + die "No $SECRETS_JSON_NAME in $PWD. + 'secrets migrate' copy-forwards a project's v1 blobs to their v2 names and + reads the project manifest to do so. cd into a project that has a manifest, + then run 'secrets migrate'. (Store-wide 'secrets migrate --finalize' comes + after every project is migrated.)" + fi + _check_manifest_file "$manifest" + local project; project=$(derive_project_name "") + local pdir="$SECRETS_DIR/$project" + + local moved=0 already=0 would=0 etype epath slug old new + while IFS=$'\t' read -r etype epath _; do + [ -n "$etype" ] || continue + # Only `properties` blobs change name in v2; dotenv and `file` are already + # in their v2 shape and never move. + [ "$etype" = "gradle-properties" ] || continue + slug=$(_secrets_files_slug "$epath") + old="$pdir/external/$slug.gradle-properties.age" + new="$pdir/external/$slug.properties.age" + [ -f "$old" ] || continue # nothing pushed yet (or already dropped) + if [ -f "$new" ]; then # idempotent: twin already exists + already=$((already + 1)) + continue + fi + if [ "$dry_run" = true ]; then + echo "would migrate: $project/external/$slug.gradle-properties.age -> $slug.properties.age" + would=$((would + 1)) + else + cp "$old" "$new" + moved=$((moved + 1)) + fi + done < <(_json_external_entries "$manifest") + + if [ "$dry_run" = true ]; then + echo "migrate --dry-run: $would blob(s) would be copy-forwarded for '$project' (writes nothing); $already already present. v1 blobs are kept until 'secrets migrate --finalize'." + return 0 + fi + if [ "$moved" -eq 0 ] && [ "$already" -eq 0 ]; then + info "Nothing to migrate for '$project' (no v1 properties blobs)." + return 0 + fi + ensure_store_protections + git -C "$SECRETS_DIR" add -A + git -C "$SECRETS_DIR" commit -m "migrate: copy-forward v2 twins for $project" >/dev/null 2>&1 || true + info "Copy-forward for '$project': $moved new v2 twin(s), $already already present. v1 blobs kept (non-destructive). Run 'secrets migrate --finalize' once every project is migrated and every machine is upgraded." +} + +# Store-wide finalize: the only destructive step. Refuses unless verify --all +# is green and every v1 properties blob has a v2 twin. Cuts a recovery tag, +# stamps the marker, then drops v1 blobs. +_migrate_finalize() { + local force="$1" + check_key + if [ "$(_store_format)" = "2" ]; then + info "Store is already format v2 — nothing to finalize." + return 0 + fi + + # (gate 1) every blob must decrypt with the current key. + info "Verifying every blob decrypts before finalizing..." + if ! _verify_all >/dev/null 2>&1; then + die "Refusing to finalize: 'secrets verify --all' is not green — a blob does not decrypt. Run 'secrets verify --all' to see which, fix it, then re-run --finalize." + fi + + # (gate 2) every v1 properties blob must have a v2 twin (project migrated). + local untwinned="" f new v1count=0 + while IFS= read -r f; do + [ -f "$f" ] || continue + v1count=$((v1count + 1)) + new="${f%.gradle-properties.age}.properties.age" + [ -f "$new" ] || untwinned="$untwinned ${f#"$SECRETS_DIR"/}"$'\n' + done < <(find "$SECRETS_DIR" -type f -name '*.gradle-properties.age') + if [ -n "$untwinned" ]; then + die "Refusing to finalize: these v1 properties blobs have no v2 twin (their project was not migrated): +$untwinned cd into each project and run 'secrets migrate', then re-run 'secrets migrate --finalize'." + fi + if [ "$v1count" -eq 0 ]; then + # No v1 blobs at all — just stamp the marker (dotenv/file-only store). + printf '2\n' > "$SECRETS_DIR/$STORE_FORMAT_FILE_NAME" + ensure_store_protections + git -C "$SECRETS_DIR" add -A + git -C "$SECRETS_DIR" commit -m "migrate: finalize store format v2" >/dev/null 2>&1 || true + git -C "$SECRETS_DIR" remote get-url origin >/dev/null 2>&1 && git -C "$SECRETS_DIR" push >/dev/null 2>&1 || true + info "Store finalized to format v2 (no v1 blobs to drop)." + return 0 + fi + + # (gate 3) operator confirms every machine is upgraded. + if [ "$force" != true ]; then + echo "Finalize will drop $v1count v1 blob(s) and stamp the store format v2." + echo "Any machine still running a pre-v2 'secrets' will stop seeing 'properties'" + echo "externals from this store until it upgrades (git pull in the secrets repo)." + printf "Confirm every machine is upgraded? Type 'yes': " + local reply="" + read -r reply < /dev/tty 2>/dev/null || read -r reply || true + [ "$reply" = "yes" ] || die "Finalize aborted — no confirmation." + fi + + # Recovery tag BEFORE any mutation: points at the pre-finalize commit (v1 + # blobs intact, no marker), so `git checkout ` fully restores v1. + local tag="pre-v2-migrate-$(git -C "$SECRETS_DIR" rev-parse --short HEAD 2>/dev/null || echo unknown)" + git -C "$SECRETS_DIR" tag "$tag" >/dev/null 2>&1 || true + + # Stamp the marker FIRST, then drop v1 blobs. If finalize crashes between + # the two, the store reads as v2 and the (verified) v2 twins serve every + # upgraded client; leftover v1 blobs are harmless orphans a re-run cleans. + printf '2\n' > "$SECRETS_DIR/$STORE_FORMAT_FILE_NAME" + while IFS= read -r f; do + [ -f "$f" ] || continue + rm -f "$f" + done < <(find "$SECRETS_DIR" -type f -name '*.gradle-properties.age') + + ensure_store_protections + git -C "$SECRETS_DIR" add -A + git -C "$SECRETS_DIR" commit -m "migrate: finalize store format v2 (drop $v1count v1 blob(s))" >/dev/null 2>&1 || true + git -C "$SECRETS_DIR" remote get-url origin >/dev/null 2>&1 && git -C "$SECRETS_DIR" push >/dev/null 2>&1 || true + info "Store finalized to format v2. Dropped $v1count v1 blob(s). Recovery tag in the store: $tag" +} + +cmd_migrate() { + resolve_store + check_initialized + local dry_run=false finalize=false force=false + while [ $# -gt 0 ]; do + case "$1" in + --dry-run) dry_run=true; shift ;; + --finalize) finalize=true; shift ;; + --yes|--force) force=true; shift ;; + -*) die "Unknown migrate flag: $1. Usage: secrets migrate [--dry-run | --finalize] [--yes]" ;; + *) die "migrate takes no project argument. Run it from inside a project (copy-forward) or use --finalize (store-wide)." ;; + esac + done + if [ "$finalize" = true ]; then + _migrate_finalize "$force" + else + _migrate_project "$dry_run" + fi +} + cmd_help() { cat << 'EOF' secrets — encrypted secret file sync between machines @@ -2087,6 +2288,8 @@ Usage: secrets rekey Re-encrypt all secrets with a new key secrets verify [project] Check the manifest against the store + decrypt every blob secrets verify --all Decrypt-test every blob in every project (integrity gate) + secrets migrate [--dry-run] Copy-forward this project's blobs to store format v2 + secrets migrate --finalize Drop v1 blobs and mark the store v2 (after verify) secrets which Show the active store, manifest, and external entries secrets where Alias for `which` secrets status Alias for `which` @@ -2256,6 +2459,7 @@ case "${1:-help}" in rm) cmd_rm "${2:-}" ;; rekey) cmd_rekey ;; verify) shift; cmd_verify "$@" ;; + migrate) shift; cmd_migrate "$@" ;; which|where|status) cmd_which ;; help|--help|-h) cmd_help ;; *) die "Unknown command: $1. Run 'secrets help' for usage." ;; diff --git a/test/manifest.bats b/test/manifest.bats index 9df5bfa..37564c7 100644 --- a/test/manifest.bats +++ b/test/manifest.bats @@ -269,7 +269,7 @@ m_file_src() { mkdir -p "$HOME/keystores"; printf 'KS\x00\x01\x02\xffDATA\n' > run jq -r '.external[] | select(.type == "file") | .path' .secrets.json [ "$output" = "~/keystores/upload.keystore" ] # stage 1: blob naming stays legacy-compatible - run bash -c "ls $SECRETS_DIR/absorbproj/external/*.gradle-properties.age" + run bash -c "ls $SECRETS_DIR/absorbproj/external/*.properties.age" [ "$status" -eq 0 ] } @@ -296,7 +296,7 @@ m_file_src() { mkdir -p "$HOME/keystores"; printf 'KS\x00\x01\x02\xffDATA\n' > run "$SECRETS_BIN" push jsonextproj [ "$status" -eq 0 ] [[ "$output" == *"Extracted 1 key"* ]] || false - run bash -c "ls $SECRETS_DIR/jsonextproj/external/*.gradle-properties.age" + run bash -c "ls $SECRETS_DIR/jsonextproj/external/*.properties.age" [ "$status" -eq 0 ] } diff --git a/test/migrate.bats b/test/migrate.bats new file mode 100644 index 0000000..b7559a6 --- /dev/null +++ b/test/migrate.bats @@ -0,0 +1,270 @@ +#!/usr/bin/env bats +# EGB-703 store-format-v2: marker, format-aware suffix, migrate (dry-run / +# copy-forward / finalize). bash 3.2: every standalone [[ ]] ends with || false. + +load test_helper + +# A v1 (legacy) store: born-v2 init, then strip the marker so it reads as v1 +# and pushes write the legacy .gradle-properties.age suffix. +make_v1_store() { + init_with_remote + rm -f "$SECRETS_DIR/.secrets-format" +} +m_gradle_src() { mkdir -p "$HOME/.gradle"; printf '%s' "$1" > "$HOME/.gradle/gradle.properties"; } +m_file_src() { mkdir -p "$HOME/keystores"; printf 'KS\x00\x01\x02\xffDATA\n' > "$HOME/keystores/upload.keystore"; } + +# ─── Format marker + format-aware suffix (increment 1) ──────────────── + +@test "init stamps the store format marker as v2 (born-v2)" { + init_with_remote + [ -f "$SECRETS_DIR/.secrets-format" ] + [ "$(cat "$SECRETS_DIR/.secrets-format")" = "2" ] +} + +@test "which prints format v2 for a born-v2 store" { + init_with_remote + create_project_dir whichv2 + run "$SECRETS_BIN" which + [ "$status" -eq 0 ] + [[ "$output" == *"format: v2"* ]] || false +} + +@test "which prints format v1 for a markerless (legacy) store" { + make_v1_store + create_project_dir whichv1 + run "$SECRETS_BIN" which + [ "$status" -eq 0 ] + [[ "$output" == *"format: v1"* ]] || false +} + +@test "push on a born-v2 store writes the properties blob as .properties.age" { + init_with_remote + m_gradle_src $'beaconClerkPkTest=pk_test_abc\n' + create_project_dir v2push + printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files + "$SECRETS_BIN" push v2push >/dev/null 2>&1 + run bash -c "ls $SECRETS_DIR/v2push/external/*.properties.age" + [ "$status" -eq 0 ] + run bash -c "ls $SECRETS_DIR/v2push/external/*.gradle-properties.age 2>/dev/null" + [ "$status" -ne 0 ] +} + +@test "push on a v1 store still writes .gradle-properties.age (back-compat)" { + make_v1_store + m_gradle_src $'beaconClerkPkTest=pk_test_abc\n' + create_project_dir v1push + printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files + "$SECRETS_BIN" push v1push >/dev/null 2>&1 + run bash -c "ls $SECRETS_DIR/v1push/external/*.gradle-properties.age" + [ "$status" -eq 0 ] +} + +@test "the format marker is committed, not gitignored" { + init_with_remote + create_project_dir markercommit + "$SECRETS_BIN" push markercommit >/dev/null 2>&1 + run bash -c "git -C $SECRETS_DIR ls-files | grep -qx .secrets-format" + [ "$status" -eq 0 ] +} + +# ─── migrate --dry-run / copy-forward (increment 2) ─────────────────── + +@test "migrate --dry-run reports the rename and writes nothing" { + make_v1_store + m_gradle_src $'beaconClerkPkTest=pk_test_abc\n' + create_project_dir dryproj + printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files + "$SECRETS_BIN" push dryproj >/dev/null 2>&1 + run "$SECRETS_BIN" migrate --dry-run + [ "$status" -eq 0 ] + [[ "$output" == *"would migrate"* ]] || false + # nothing written + run bash -c "ls $SECRETS_DIR/dryproj/external/*.properties.age 2>/dev/null" + [ "$status" -ne 0 ] + # marker still absent (store still v1) + [ ! -f "$SECRETS_DIR/.secrets-format" ] +} + +@test "migrate --dry-run on a dotenv-only project reports nothing to migrate" { + make_v1_store + create_project_dir dotenvonly + "$SECRETS_BIN" push dotenvonly >/dev/null 2>&1 + run "$SECRETS_BIN" migrate --dry-run + [ "$status" -eq 0 ] + [[ "$output" == *"0 blob(s) would be copy-forwarded"* ]] || false +} + +@test "migrate copy-forward creates the v2 twin and keeps the v1 blob (byte-identical)" { + make_v1_store + m_gradle_src $'beaconClerkPkTest=pk_test_abc\n' + create_project_dir cfproj + printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files + "$SECRETS_BIN" push cfproj >/dev/null 2>&1 + local old; old=$(ls "$SECRETS_DIR/cfproj/external/"*.gradle-properties.age) + run "$SECRETS_BIN" migrate + [ "$status" -eq 0 ] + local new; new=$(ls "$SECRETS_DIR/cfproj/external/"*.properties.age) + [ -f "$old" ] # v1 kept (non-destructive) + [ -f "$new" ] # v2 twin written + cmp -s "$old" "$new" # byte-identical ciphertext copy +} + +@test "migrate copy-forward is idempotent" { + make_v1_store + m_gradle_src $'beaconClerkPkTest=pk_test_abc\n' + create_project_dir idemproj + printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files + "$SECRETS_BIN" push idemproj >/dev/null 2>&1 + "$SECRETS_BIN" migrate >/dev/null 2>&1 + run "$SECRETS_BIN" migrate + [ "$status" -eq 0 ] + [[ "$output" == *"1 already present"* ]] || false + run bash -c "ls $SECRETS_DIR/idemproj/external/*.properties.age | wc -l | tr -d ' '" + [ "$output" = "1" ] +} + +@test "migrate with no manifest in cwd dies with a directed message" { + make_v1_store + local dir="$WORK_DIR/nomanifest"; mkdir -p "$dir"; cd "$dir" + run "$SECRETS_BIN" migrate + [ "$status" -eq 1 ] + [[ "$output" == *".secrets.json"* ]] || false +} + +@test "migrate on an already-v2 store is a no-op" { + init_with_remote + create_project_dir alreadyv2 + run "$SECRETS_BIN" migrate + [ "$status" -eq 0 ] + [[ "$output" == *"already format v2"* ]] || false +} + +@test "migrate unknown flag dies with usage" { + init_with_remote + create_project_dir mgflag + run "$SECRETS_BIN" migrate --bogus + [ "$status" -eq 1 ] + [[ "$output" == *"Unknown migrate flag"* ]] || false +} + +@test "migrate leaves dotenv and file blobs untouched" { + make_v1_store + m_gradle_src $'beaconClerkPkTest=pk_test_abc\n' + m_file_src + create_project_dir mixproj + printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\nfile ~/keystores/upload.keystore\n' > .secrets-files + "$SECRETS_BIN" push mixproj >/dev/null 2>&1 + local envblob; envblob=$(ls "$SECRETS_DIR/mixproj/".env.age) + local fileblob; fileblob=$(ls "$SECRETS_DIR/mixproj/external/"*.file.age) + local envsum; envsum=$(cksum "$envblob") + local filesum; filesum=$(cksum "$fileblob") + "$SECRETS_BIN" migrate >/dev/null 2>&1 + [ "$(cksum "$envblob")" = "$envsum" ] # dotenv blob unchanged + [ "$(cksum "$fileblob")" = "$filesum" ] # file blob unchanged +} + +# ─── migrate --finalize (increment 3) ───────────────────────────────── + +@test "finalize refuses when verify --all is not green" { + make_v1_store + m_gradle_src $'beaconClerkPkTest=pk_test_abc\n' + create_project_dir failverify + printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files + "$SECRETS_BIN" push failverify >/dev/null 2>&1 + "$SECRETS_BIN" migrate >/dev/null 2>&1 + # corrupt the v2 twin so verify --all fails + printf 'garbage' > "$SECRETS_DIR/failverify/external/"*.properties.age + run "$SECRETS_BIN" migrate --finalize --yes + [ "$status" -eq 1 ] + [[ "$output" == *"not green"* ]] || false + # marker not stamped; v1 blob still present + [ ! -f "$SECRETS_DIR/.secrets-format" ] + run bash -c "ls $SECRETS_DIR/failverify/external/*.gradle-properties.age" + [ "$status" -eq 0 ] +} + +@test "finalize refuses an un-twinned v1 blob (project not migrated)" { + make_v1_store + m_gradle_src $'beaconClerkPkTest=pk_test_abc\n' + create_project_dir untwinned + printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files + "$SECRETS_BIN" push untwinned >/dev/null 2>&1 + # do NOT migrate — leave the v1 blob with no twin + run "$SECRETS_BIN" migrate --finalize --yes + [ "$status" -eq 1 ] + [[ "$output" == *"no v2 twin"* ]] || false + run bash -c "ls $SECRETS_DIR/untwinned/external/*.gradle-properties.age" + [ "$status" -eq 0 ] +} + +@test "finalize green path drops v1, keeps v2, stamps the marker" { + make_v1_store + m_gradle_src $'beaconClerkPkTest=pk_test_abc\n' + create_project_dir finproj + printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files + "$SECRETS_BIN" push finproj >/dev/null 2>&1 + "$SECRETS_BIN" migrate >/dev/null 2>&1 + run "$SECRETS_BIN" migrate --finalize --yes + [ "$status" -eq 0 ] + [ "$(cat "$SECRETS_DIR/.secrets-format")" = "2" ] + run bash -c "ls $SECRETS_DIR/finproj/external/*.properties.age" + [ "$status" -eq 0 ] + run bash -c "ls $SECRETS_DIR/finproj/external/*.gradle-properties.age 2>/dev/null" + [ "$status" -ne 0 ] +} + +@test "finalize cuts a recovery tag before deleting v1 blobs" { + make_v1_store + m_gradle_src $'beaconClerkPkTest=pk_test_abc\n' + create_project_dir tagproj + printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files + "$SECRETS_BIN" push tagproj >/dev/null 2>&1 + "$SECRETS_BIN" migrate >/dev/null 2>&1 + "$SECRETS_BIN" migrate --finalize --yes >/dev/null 2>&1 + local tag; tag=$(git -C "$SECRETS_DIR" tag | grep '^pre-v2-migrate-') + [ -n "$tag" ] + # the tagged commit still contains the v1 blob (tag cut before delete) + run bash -c "git -C $SECRETS_DIR ls-tree -r --name-only $tag | grep -q gradle-properties.age" + [ "$status" -eq 0 ] +} + +@test "finalize without --yes aborts when not confirmed" { + make_v1_store + m_gradle_src $'beaconClerkPkTest=pk_test_abc\n' + create_project_dir confproj + printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files + "$SECRETS_BIN" push confproj >/dev/null 2>&1 + "$SECRETS_BIN" migrate >/dev/null 2>&1 + run bash -c "echo '' | $SECRETS_BIN migrate --finalize" + [ "$status" -eq 1 ] + [[ "$output" == *"aborted"* ]] || false + [ ! -f "$SECRETS_DIR/.secrets-format" ] +} + +@test "v1 client still reads during the migration window (after copy-forward, before finalize)" { + make_v1_store + m_gradle_src $'beaconClerkPkTest=pk_test_abc\n' + create_project_dir windowproj + printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files + "$SECRETS_BIN" push windowproj >/dev/null 2>&1 + "$SECRETS_BIN" migrate >/dev/null 2>&1 + # store is still v1 (markerless); pull uses the .gradle-properties.age blob + rm "$HOME/.gradle/gradle.properties" + run "$SECRETS_BIN" pull windowproj + [ "$status" -eq 0 ] + grep -q '^beaconClerkPkTest=pk_test_abc$' "$HOME/.gradle/gradle.properties" +} + +@test "post-finalize pull reads the v2 blob" { + make_v1_store + m_gradle_src $'beaconClerkPkTest=pk_test_abc\n' + create_project_dir postfin + printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files + "$SECRETS_BIN" push postfin >/dev/null 2>&1 + "$SECRETS_BIN" migrate >/dev/null 2>&1 + "$SECRETS_BIN" migrate --finalize --yes >/dev/null 2>&1 + rm "$HOME/.gradle/gradle.properties" + run "$SECRETS_BIN" pull postfin + [ "$status" -eq 0 ] + grep -q '^beaconClerkPkTest=pk_test_abc$' "$HOME/.gradle/gradle.properties" +} diff --git a/test/secrets.bats b/test/secrets.bats index 5537a03..a75c0e6 100644 --- a/test/secrets.bats +++ b/test/secrets.bats @@ -1081,7 +1081,7 @@ gradle_project() { run "$SECRETS_BIN" push gproj [ "$status" -eq 0 ] [[ "$output" == *"Extracted 2 key"* ]] || false - run bash -c "ls $SECRETS_DIR/gproj/external/*.gradle-properties.age" + run bash -c "ls $SECRETS_DIR/gproj/external/*.properties.age" [ "$status" -eq 0 ] } @@ -1368,7 +1368,7 @@ gradle_project() { run "$SECRETS_BIN" push -w [ "$status" -eq 0 ] # External blob pushed exactly once (not once per workspace) - run bash -c "ls $SECRETS_DIR/mono/external/*.gradle-properties.age 2>/dev/null | wc -l | tr -d ' '" + run bash -c "ls $SECRETS_DIR/mono/external/*.properties.age 2>/dev/null | wc -l | tr -d ' '" [ "$output" = "1" ] }