diff --git a/docs/superpowers/plans/2026-06-08-egb-710-migrate-guided-flow.md b/docs/superpowers/plans/2026-06-08-egb-710-migrate-guided-flow.md new file mode 100644 index 0000000..b552428 --- /dev/null +++ b/docs/superpowers/plans/2026-06-08-egb-710-migrate-guided-flow.md @@ -0,0 +1,447 @@ +# EGB-710: `secrets migrate` guided flow (manifest-free copy-forward + `--status` survey) Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Replace the dead-end "No .secrets.json — cd into a project that has a manifest" error in `secrets migrate` with a manifest-free copy-forward that just works, plus a `secrets migrate --status` survey that tells the operator exactly which projects still need migrating. + +**Architecture:** The per-project copy-forward step (`_migrate_project`) currently reads `$PWD/.secrets.json` to find `properties` externals, then looks for their v1 blobs. This makes it die on a legacy `.secrets-files`-only project (the real EGB-710 repro) and — worse — silently skips any store blob the manifest doesn't declare, leaving it un-twinned so `--finalize` later refuses it. The fix: enumerate the **store** (`$SECRETS_DIR//external/*.gradle-properties.age`) directly and copy each to its `.properties.age` twin via suffix-swap — the *exact* logic `_migrate_finalize` already uses. No manifest needed, dotenv/file-only projects become a clean no-op, and migrate twins precisely the blobs finalize will demand twins for. A new `--status` mode walks every project dir and reports per-project readiness + whether the store is finalize-ready. + +**Tech Stack:** Single bash 3.2 script (`secrets`), `age`, `git`, `jq` (unaffected here). Tests: `bats-core` (`test/migrate.bats`). Every standalone `[[ ]]` assertion ends with `|| false` (bash 3.2 ERR-trap gotcha). + +--- + +## Background facts (verified against HEAD 633d19e) + +- `_migrate_project()` lives at `secrets:2135`; it `die`s at `secrets:2142-2148` on missing `$PWD/.secrets.json`, then iterates `_json_external_entries "$manifest"` filtered to `gradle-properties` (`secrets:2154-2174`). +- `_migrate_finalize()` (`secrets:2196`) already enumerates blobs manifest-free: `find "$SECRETS_DIR" -type f -name '*.gradle-properties.age'` and derives the twin as `new="${f%.gradle-properties.age}.properties.age"` (`secrets:2212-2217`). The per-project step will mirror this, scoped to one project dir. +- `cmd_migrate()` flag parser: `secrets:2267-2285`. +- `derive_project_name ""` (`secrets:101`) needs no manifest — it uses the git remote basename or `basename "$PWD"`. +- `info()`/`die()`: `secrets:36-37`. `ensure_store_protections` is called post-write in the existing copy-forward. +- Output-contract strings existing tests depend on (must be preserved): `"would migrate"` (migrate.bats:80), `"0 blob(s) would be copy-forwarded"` (migrate.bats:94), `"1 already present"` (migrate.bats:121, idempotent re-run prints `$already already present`), `"already format v2"` (migrate.bats:139). +- The one test that codifies the OLD dead-end — `"migrate with no manifest in cwd dies with a directed message"` (migrate.bats:126) — is the behavior we are intentionally changing; it gets rewritten in Task 1. +- No test asserts the non-dry-run "nothing to migrate" wording (grep confirmed), so that message is free to change. We keep the substring `no v1 properties blobs` regardless for safety. +- VERSION is `0.6.0.1`; bump to `0.6.1.0` (new subcommand flag + behavior change). `MANIFEST_VERSION` stays `2` (no schema change). + +## File structure + +- Modify: `secrets` — rewrite `_migrate_project` (`secrets:2135-2191`), add `_migrate_status`, extend `cmd_migrate` flag parsing + dispatch (`secrets:2267-2285`), update `cmd_help` migrate lines (`secrets:2308-2309`). +- Modify: `test/migrate.bats` — rewrite the dead-end test; add manifest-free, finalize-consistency, and `--status` tests. +- Modify: `CLAUDE.md` — update the migrate paragraph (Architecture → Store format) to note manifest-free copy-forward + `--status`. +- Modify: `README.md` — migrate usage/help. +- Modify: `VERSION` → `0.6.1.0`; `CHANGELOG.md` — new entry. + +--- + +## Task 1: Manifest-free per-project copy-forward + +**Files:** +- Modify: `secrets` — `_migrate_project()` (`secrets:2135-2191`) +- Test: `test/migrate.bats` + +- [ ] **Step 1: Write the failing test — migrate works with no `.secrets.json` (the EGB-710 repro)** + +Add to `test/migrate.bats` (after the existing copy-forward tests, ~line 124): + +```bash +@test "migrate copy-forwards a v1 properties blob with no .secrets.json (manifest-free)" { + # The EGB-710 repro: a legacy project has a v1 properties blob in the store + # but no .secrets.json (it predates the manifest). migrate must NOT dead-end. + make_v1_store + m_gradle_src $'beaconClerkPkTest=pk_test_abc\n' + create_project_dir nomanifestblob + printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files + "$SECRETS_BIN" push nomanifestblob >/dev/null 2>&1 + rm -f .secrets.json # simulate a pre-manifest project + run "$SECRETS_BIN" migrate + [ "$status" -eq 0 ] + run bash -c "ls $SECRETS_DIR/nomanifestblob/external/*.properties.age" + [ "$status" -eq 0 ] # v2 twin written despite no manifest + run bash -c "ls $SECRETS_DIR/nomanifestblob/external/*.gradle-properties.age" + [ "$status" -eq 0 ] # v1 kept (non-destructive) +} +``` + +- [ ] **Step 2: Run it to confirm it fails** + +Run: `bats test/migrate.bats -f "manifest-free"` +Expected: FAIL — current code `die`s with "No .secrets.json in …" (status 1), so the first `[ "$status" -eq 0 ]` fails. + +- [ ] **Step 3: Rewrite `_migrate_project` to enumerate the store, not the manifest** + +Replace the body of `_migrate_project()` (`secrets:2135-2191`, from `_migrate_project() {` through its closing `}`) with: + +```bash +_migrate_project() { + local dry_run="$1" + if [ "$(_store_format)" = "2" ]; then + info "Store is already format v2 — nothing to migrate." + return 0 + fi + local project; project=$(derive_project_name "") + local pdir="$SECRETS_DIR/$project" + + # Source of truth for the copy-forward is the STORE, not a project manifest. + # Every v1 properties blob is a `*.gradle-properties.age` file whose v2 twin + # is the same name with the `.properties.age` suffix (the only on-disk change + # v2 makes). Enumerating the store — exactly as `_migrate_finalize` does — + # means migrate twins precisely the blobs finalize will demand twins for, with + # no manifest dependency. This is why a legacy `.secrets-files`-only project + # (no `.secrets.json` yet) migrates cleanly instead of dead-ending, and why a + # store blob the manifest no longer declares still gets a twin. + local moved=0 already=0 would=0 v1count=0 f new + while IFS= read -r f; do + [ -f "$f" ] || continue + v1count=$((v1count + 1)) + new="${f%.gradle-properties.age}.properties.age" + if [ -f "$new" ]; then + already=$((already + 1)) + continue + fi + if [ "$dry_run" = true ]; then + echo "would migrate: ${f#"$SECRETS_DIR"/} -> $(basename "$new")" + would=$((would + 1)) + else + cp "$f" "$new" + moved=$((moved + 1)) + fi + done < <(find "$pdir/external" -type f -name '*.gradle-properties.age' 2>/dev/null) + + if [ "$dry_run" = true ]; then + echo "migrate --dry-run: $would blob(s) would be copy-forwarded for '$project' (writes nothing); $already already present. v1 blobs are kept until 'secrets migrate --finalize'." + return 0 + fi + if [ "$moved" -eq 0 ] && [ "$already" -eq 0 ]; then + info "Nothing to migrate for '$project' — no v1 properties blobs in the store (already v2-shaped). If you expected one, run 'secrets push' first, then re-run 'secrets migrate'." + return 0 + fi + ensure_store_protections + git -C "$SECRETS_DIR" add -A + git -C "$SECRETS_DIR" commit -m "migrate: copy-forward v2 twins for $project" >/dev/null 2>&1 || true + # Push the twins so a --finalize on another machine sees them (finalize + # refuses any v1 blob without a twin). Mirrors push/rekey's push behavior. + git -C "$SECRETS_DIR" remote get-url origin >/dev/null 2>&1 && git -C "$SECRETS_DIR" push >/dev/null 2>&1 || true + info "Copy-forward for '$project': $moved new v2 twin(s), $already already present. v1 blobs kept (non-destructive). Run 'secrets migrate --finalize' once every project is migrated and every machine is upgraded." +} +``` + +Notes for the implementer: +- This removes the only in-script caller of `_check_manifest_file`/`_json_external_entries` *inside migrate*; both remain defined and used by push/pull/verify, so do **not** delete them. +- `${f#"$SECRETS_DIR"/}` keeps the dry-run line's store-relative form (matches the old `$project/external/...` style closely enough; the test only checks the substring `would migrate`). +- The `moved==0 && already==0` branch keeps the substring `no v1 properties blobs`. The idempotent re-run path (`moved==0, already>0`) falls through to the final `info` printing `$already already present`, preserving the `1 already present` contract. + +- [ ] **Step 4: Run the new test + the full migrate suite** + +Run: `bats test/migrate.bats` +Expected: the new "manifest-free" test PASSES; existing dry-run/copy-forward/idempotent/finalize tests still PASS; the "migrate with no manifest in cwd dies" test (migrate.bats:126) now FAILS (we fix it in Step 5). + +- [ ] **Step 5: Update the test that codified the old dead-end** + +Replace the test at `test/migrate.bats:126-132` (`"migrate with no manifest in cwd dies with a directed message"`) with: + +```bash +@test "migrate in a project with no manifest and no store blobs is a clean no-op" { + make_v1_store + local dir="$WORK_DIR/nomanifest"; mkdir -p "$dir"; cd "$dir" + run "$SECRETS_BIN" migrate + [ "$status" -eq 0 ] + [[ "$output" == *"no v1 properties blobs"* ]] || false +} +``` + +- [ ] **Step 6: Run the full suite to confirm green** + +Run: `bats test/migrate.bats` +Expected: all PASS. + +- [ ] **Step 7: Commit** + +```bash +git add secrets test/migrate.bats +git commit -m "fix: migrate copy-forward is manifest-free, no dead-end on legacy projects (EGB-710)" +``` + +--- + +## Task 2: Finalize-consistency regression test (store blob not in manifest) + +**Files:** +- Test: `test/migrate.bats` + +This proves the latent-bug fix: the old manifest-driven migrate skipped store blobs the manifest didn't declare, leaving them un-twinned so `--finalize` refused them. The rewrite twins them. + +- [ ] **Step 1: Write the test** + +Add to `test/migrate.bats`: + +```bash +@test "migrate twins a store blob even when the manifest no longer declares it" { + make_v1_store + m_gradle_src $'beaconClerkPkTest=pk_test_abc\n' + create_project_dir staleblob + printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files + "$SECRETS_BIN" push staleblob >/dev/null 2>&1 + # The blob is now in the store. Drop the external from the project's manifest + # entirely (and remove the legacy file) so NO manifest declares it. + printf '{"version":2,"dotenv":[".env",".env.staging"]}\n' > .secrets.json + rm -f .secrets-files + run "$SECRETS_BIN" migrate + [ "$status" -eq 0 ] + run bash -c "ls $SECRETS_DIR/staleblob/external/*.properties.age" + [ "$status" -eq 0 ] # twinned despite not being declared anywhere +} +``` + +- [ ] **Step 2: Run it** + +Run: `bats test/migrate.bats -f "no longer declares"` +Expected: PASS (the Task 1 rewrite already makes this green — this test guards against regressing back to manifest-driven enumeration). + +- [ ] **Step 3: Commit** + +```bash +git add test/migrate.bats +git commit -m "test: migrate twins undeclared store blobs (finalize-consistency, EGB-710)" +``` + +--- + +## Task 3: `secrets migrate --status` survey + +**Files:** +- Modify: `secrets` — add `_migrate_status()`; extend `cmd_migrate` (`secrets:2267-2285`) +- Test: `test/migrate.bats` + +- [ ] **Step 1: Write the failing tests** + +Add to `test/migrate.bats`: + +```bash +@test "migrate --status flags a project that needs migrating" { + make_v1_store + m_gradle_src $'beaconClerkPkTest=pk_test_abc\n' + create_project_dir needsmig + printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files + "$SECRETS_BIN" push needsmig >/dev/null 2>&1 # v1 blob, no twin yet + run "$SECRETS_BIN" migrate --status + [ "$status" -ne 0 ] # not finalize-ready + [[ "$output" == *"needsmig"* ]] || false + [[ "$output" == *"NEEDS MIGRATE"* ]] || false + [[ "$output" == *"Not finalize-ready"* ]] || false +} + +@test "migrate --status reports finalize-ready once every blob is twinned" { + make_v1_store + m_gradle_src $'beaconClerkPkTest=pk_test_abc\n' + create_project_dir readymig + printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files + "$SECRETS_BIN" push readymig >/dev/null 2>&1 + "$SECRETS_BIN" migrate >/dev/null 2>&1 # create the twin + run "$SECRETS_BIN" migrate --status + [ "$status" -eq 0 ] + [[ "$output" == *"Finalize-ready"* ]] || false +} + +@test "migrate --status on an already-v2 store says nothing to do" { + init_with_remote + create_project_dir v2status + run "$SECRETS_BIN" migrate --status + [ "$status" -eq 0 ] + [[ "$output" == *"v2"* ]] || false +} +``` + +- [ ] **Step 2: Run them to confirm they fail** + +Run: `bats test/migrate.bats -f "status"` +Expected: FAIL — `--status` is an unknown flag today (`die "Unknown migrate flag: --status…"`, status 1), so the assertions fail. + +- [ ] **Step 3: Add `_migrate_status` (place it just before `_migrate_finalize`, ~`secrets:2195`)** + +```bash +# Read-only survey: walk every project dir in the store and report each one's +# v2-readiness from the blobs on disk (no manifest, no decryption). Exits +# non-zero when any v1 properties blob lacks a v2 twin (i.e. the store is not +# yet finalize-ready) so it can gate scripting, mirroring `verify`'s posture. +_migrate_status() { + if [ "$(_store_format)" = "2" ]; then + info "Store format: v2 (finalized) — nothing to migrate." + return 0 + fi + echo "Store format: v1 (not finalized). Per-project migration status:" + local any_untwinned=0 dir project v1 untwinned f new + for dir in "$SECRETS_DIR"/*/; do + [ -d "$dir" ] || continue + project=$(basename "$dir") + case "$project" in .*) continue ;; esac + v1=0; untwinned=0 + while IFS= read -r f; do + [ -f "$f" ] || continue + v1=$((v1 + 1)) + new="${f%.gradle-properties.age}.properties.age" + [ -f "$new" ] || untwinned=$((untwinned + 1)) + done < <(find "$dir" -type f -name '*.gradle-properties.age' 2>/dev/null) + if [ "$v1" -eq 0 ]; then + echo " $project: v2-ready (no v1 properties blobs)" + elif [ "$untwinned" -eq 0 ]; then + echo " $project: migrated ($v1 v1 blob(s), all twinned)" + else + echo " $project: NEEDS MIGRATE ($untwinned of $v1 v1 blob(s) un-twinned) — cd into the project and run 'secrets migrate'" + any_untwinned=1 + fi + done + echo + if [ "$any_untwinned" -eq 1 ]; then + echo "Not finalize-ready: migrate the projects marked NEEDS MIGRATE, then run 'secrets migrate --finalize'." + return 1 + fi + echo "Finalize-ready: every v1 properties blob has a v2 twin. Run 'secrets migrate --finalize' once every machine is upgraded." + return 0 +} +``` + +- [ ] **Step 4: Wire `--status` into `cmd_migrate`** + +In `cmd_migrate()` (`secrets:2267`), add the `status` local and parse + dispatch. Replace: + +```bash + local dry_run=false finalize=false force=false + while [ $# -gt 0 ]; do + case "$1" in + --dry-run) dry_run=true; shift ;; + --finalize) finalize=true; shift ;; + --yes) force=true; shift ;; + -*) die "Unknown migrate flag: $1. Usage: secrets migrate [--dry-run | --finalize] [--yes]" ;; + *) die "migrate takes no project argument. Run it from inside a project (copy-forward) or use --finalize (store-wide)." ;; + esac + done + if [ "$finalize" = true ]; then + _migrate_finalize "$force" + else + _migrate_project "$dry_run" + fi +``` + +with: + +```bash + local dry_run=false finalize=false force=false status=false + while [ $# -gt 0 ]; do + case "$1" in + --dry-run) dry_run=true; shift ;; + --finalize) finalize=true; shift ;; + --status) status=true; shift ;; + --yes) force=true; shift ;; + -*) die "Unknown migrate flag: $1. Usage: secrets migrate [--dry-run | --status | --finalize] [--yes]" ;; + *) die "migrate takes no project argument. Run it from inside a project (copy-forward), or use --status / --finalize (store-wide)." ;; + esac + done + if [ "$status" = true ]; then + _migrate_status + elif [ "$finalize" = true ]; then + _migrate_finalize "$force" + else + _migrate_project "$dry_run" + fi +``` + +- [ ] **Step 5: Run the status tests + full suite** + +Run: `bats test/migrate.bats` +Expected: all PASS. + +- [ ] **Step 6: Commit** + +```bash +git add secrets test/migrate.bats +git commit -m "feat: secrets migrate --status surveys per-project v2 readiness (EGB-710)" +``` + +--- + +## Task 4: Docs, help text, version, changelog + +**Files:** +- Modify: `secrets` — `cmd_help` (`secrets:2308-2309`) +- Modify: `CLAUDE.md`, `README.md`, `VERSION`, `CHANGELOG.md` + +- [ ] **Step 1: Update `cmd_help` migrate lines** + +In `cmd_help()` replace the two migrate lines (`secrets:2308-2309`): + +``` + secrets migrate [--dry-run] Copy-forward this project's blobs to store format v2 + secrets migrate --finalize Drop v1 blobs and mark the store v2 (after verify) +``` + +with: + +``` + secrets migrate [--dry-run] Copy-forward this project's v1 blobs to store format v2 + secrets migrate --status Survey every project's v2 readiness (finalize gate) + secrets migrate --finalize Drop v1 blobs and mark the store v2 (after verify) +``` + +- [ ] **Step 2: Update `CLAUDE.md` migrate paragraph** + +In the "Store format (EGB-677 stage 2 / EGB-703)" bullet, update the migration description: copy-forward is now **manifest-free** — `secrets migrate` enumerates the project's `*.gradle-properties.age` store blobs directly (same source of truth as `--finalize`), so a legacy `.secrets-files`-only project migrates without a `.secrets.json` and no store blob is left un-twinned. Add `secrets migrate --status` to the workflow line as the read-only survey that reports per-project readiness and gates `--finalize`. Reference EGB-710. + +- [ ] **Step 3: Update `README.md`** + +Find the migrate section/help block and add the `--status` line and the manifest-free note, mirroring the help text. + +- [ ] **Step 4: Bump `VERSION`** + +Set `VERSION` to `0.6.1.0`. + +- [ ] **Step 5: Add `CHANGELOG.md` entry** + +Insert above `## [0.6.0.1]`: + +```markdown +## [0.6.1.0] - 2026-06-08 + +### Changed + +- **`secrets migrate` copy-forward is now manifest-free (EGB-710)** — the + per-project step enumerates the store's `*.gradle-properties.age` blobs + directly (the same source of truth `--finalize` uses) instead of reading + `.secrets.json`. A legacy `.secrets-files`-only project now migrates cleanly + instead of dead-ending with "No .secrets.json", and a store blob the manifest + no longer declares still gets a v2 twin (so `--finalize` won't refuse it). + Running migrate in a project with no v1 properties blobs is a clean no-op. + +### Added + +- **`secrets migrate --status`** — a read-only survey that walks every project + in the store and reports its v2 readiness (v2-ready / migrated / NEEDS + MIGRATE), then whether the store as a whole is finalize-ready. Exits non-zero + while any v1 blob is un-twinned, so it can gate the path to `--finalize`. +``` + +- [ ] **Step 6: Run the full bats suite (all files)** + +Run: `bats test/` +Expected: all PASS (secrets.bats + manifest.bats + migrate.bats). Confirm the count went up by the tests added here. + +- [ ] **Step 7: Commit** + +```bash +git add secrets CLAUDE.md README.md VERSION CHANGELOG.md +git commit -m "docs: migrate --status + manifest-free copy-forward; bump 0.6.1.0 (EGB-710)" +``` + +--- + +## Self-review against the EGB-710 spec + +- **AC: dotenv-only project exits 0 "already v2-shaped"** → Task 1 (`moved==0 && already==0` branch, substring `no v1 properties blobs`). Test: "no manifest and no store blobs is a clean no-op" (Task 1 Step 5). +- **AC: `.secrets-files`-only project with v1 blobs migrates instead of the generic error** → the manifest-free rewrite makes it *just migrate* (better than guiding to push first). Test: "manifest-free" (Task 1 Step 1). Note the rewrite supersedes the ticket's "offer to run absorb then migrate" branch — migrate no longer needs the manifest, so there's nothing to prompt for; the only destructive step (`--finalize`) keeps its existing `/dev/tty` confirmation. +- **AC: interactive prompts read `/dev/tty`, degrade non-interactively, bash 3.2** → no new prompt is introduced (copy-forward is non-destructive); `--finalize`'s existing `/dev/tty` confirm is untouched. `--status` is pure read-only output. All new code is bash 3.2 (no associative arrays, `[[ ]]` only in tests with `|| false`). +- **AC: no behavior change to copy-forward/finalize safety gates** → finalize untouched; copy-forward stays non-destructive (v1 kept, commit + push twins). Verified by the unchanged finalize tests (migrate.bats:168-214). +- **AC: bats coverage per branch** → Tasks 1-3 add: manifest-free migrate, no-op no-blobs, undeclared-blob twinning, `--status` needs-migrate / finalize-ready / already-v2. +- **Stretch: `--status` survey** → Task 3, delivered. +- **Sequencing: independent of the EGB-709 gate** → confirmed; this only touches migrate ergonomics and helps operators *reach* all-v2. EGB-703 safety posture (recovery tag, verify-green gate, twin-before-drop) is preserved. + +## Operator-local follow-up (not part of this plan) + +This repo's `.ship-policy.json` opts out of AI security/red-team passes; before any PR, ask the operator to run `./test/run-security.sh` and complete the SIGNOFF. After landing, the operator can re-run their real-store migration for `onefinalmessage` et al. (`secrets migrate` per project → `secrets migrate --finalize`), which is the path to clearing the EGB-709 v2 gate.