diff --git a/.ship-policy.json b/.ship-policy.json deleted file mode 100644 index 034532e..0000000 --- a/.ship-policy.json +++ /dev/null @@ -1,15 +0,0 @@ -{ - "version": 1, - "ai_review": { - "adversarial": false, - "red_team": false, - "security_specialist": false, - "reason": "This repo ships defensive security regression tests with attack-payload fixtures (shell injection, path traversal, symlink rails). AI-driven adversarial, red-team, and security-specialist review triggers Anthropic Usage Policy blocks (real-time cyber safeguards on hosted models). Do not dispatch those subagents or prompts for this repository." - }, - "manual_security_signoff": { - "script": "./test/run-security.sh", - "signoff_file": ".gstack/security-signoff.json", - "required_before_ship": true, - "instructions": "A human operator runs the script locally, reviews output, and types SIGNOFF. Agents must not run the security suite or substitute AI review for sign-off." - } -} diff --git a/CHANGELOG.md b/CHANGELOG.md index 4bee6fb..214b95c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,395 +5,6 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to a four-digit MAJOR.MINOR.PATCH.MICRO version scheme. -## [0.7.7.0] - 2026-09-08 - -### Fixed - -- **pnpm monorepos discovered no workspace secrets (EGB-1232)** — both - workspace call sites resolved patterns from `package.json`'s `workspaces` - key only, which pnpm does not use (it declares `packages:` in - `pnpm-workspace.yaml`). `secrets push -w` refused outright; plain `secrets - push` failed *silently* — `_maybe_workspace_env_files` returned 0 the moment - the key was absent, so the auto-discovery that exists to cover push's - root-only scan was inert on every pnpm repo and printed "Nothing new to - add", indistinguishable from a repo that genuinely had nothing. Workspace - patterns now resolve through one shared source that falls back to - `pnpm-workspace.yaml`. -- **yarn's object `workspaces` form was never expanded (EGB-1232)** — the - filter `.workspaces // .workspaces.packages | .[]` short-circuits on yarn's - truthy object, so `.workspaces.packages` was never evaluated and `.[]` - iterated the object's values, yielding the pattern array itself as a single - token. Only npm's array form worked. Now type-aware, handling npm's array, - yarn's object, and absent/null alike. - -### Added - -- **`pnpm-workspace.yaml` support** — the `packages:` block sequence is read - without a YAML dependency: block form only, stopping at the next top-level - key so pnpm 10's `onlyBuiltDependencies:`/`catalog:` cannot leak in as glob - patterns, with quote and inline-comment handling and a symlink refusal. - `package.json` wins when it declares workspaces; `pnpm-workspace.yaml` is the - fallback. jq is now required only when `package.json` is the source, so a - pnpm-only repo resolves workspaces jq-free. -- **Workspace patterns are validated before glob expansion** — no absolute - paths, `..` traversal, shell metacharacters, or whitespace reach the - unquoted expansion; pnpm `!` negations are skipped. Same conservative rail - as `.secrets-store` / `.secrets-files`. -- **A monorepo-shaped root that resolves no workspaces now says so** — if a - `pnpm-workspace.yaml` or `packages/` directory is present but no workspace - packages can be read, `push` warns on stderr and points at `secrets add`, - instead of returning in silence. `secrets push -w`'s error now names - `pnpm-workspace.yaml` when that is the file present, rather than blaming a - `package.json` the repo may not use for workspaces. - -## [0.7.6.0] - 2026-09-08 - -### Added - -- **`secrets sync` — reconcile a diverged store (EGB-1231)** — the store is a - git repo, and once a clone was both ahead and behind its remote the CLI had - no way out: `push` demanded a fast-forward and pointed at `pull`, which - could not fast-forward either, so the advice looped and recovery meant - hand-running git next to a directory of encrypted blobs. `secrets sync` - fetches, stashes uncommitted blob edits, rebases local commits onto the - remote, restores the stash, and then asks before publishing local commits to - the shared store. `--yes` skips the prompt (scripts/CI); `--dry-run` reports - ahead/behind/dirty and what would happen, changing nothing. Deliberately - non-destructive: no merge, no force-push, no `reset --hard`, no `stash - drop`. A rebase conflict aborts, restores the stash, names the conflicting - files, and leaves the store exactly as found. -- **Store state in `secrets which` (EGB-1231)** — a new `remote:` line reports - the store's `ahead N, behind N, N modified` (or `up to date`) against its - upstream, with a `(run: secrets sync)` hint when there is anything to - reconcile. Offline-safe (reports against the last fetch, never reaches the - network) and silent for a local-only store or one with no upstream. - -### Fixed - -- **`secrets pull` no longer fails silently when the store can't sync - (EGB-1230)** — the store sync was `git pull >/dev/null 2>&1` under `set -euo - pipefail`, so a store that could not fast-forward killed the script at that - line with git's exit 128 and *nothing* on stdout or stderr. The user saw a - banner, no restored files, and no reason — indistinguishable from a project - with nothing to pull, and easy to lose entirely in a pipeline. The sync is - now guarded, git's output is captured and surfaced as the diagnosis, and the - error names the store path and points at `secrets sync`. - -### Changed - -- **`secrets pull`'s store sync is now fast-forward only**, matching the push - path. A plain `git pull` could quietly manufacture a merge commit in the - store; divergence is now resolved in exactly one place — `secrets sync`. -- **The push path's dead-end advice** ("Run 'secrets pull' first, then retry - push") now points at `secrets sync` and includes git's own output. - -## [0.7.5.0] - 2026-06-24 - -### Added - -- **Multi-recipient age encryption (EGB-283)** — a store-scoped, committed - `recipients.txt` (age `-R` format, with `# name` comment lines) lets one - store encrypt every blob to N age public keys — one per team member. - `secrets recipients add [--name N]` adds a key and immediately - re-encrypts the whole store; `secrets recipients rm [--yes]` - removes one and re-encrypts; `secrets recipients list` shows the current - set (or a note that the store is still single-key). A new `secrets - reencrypt` command re-encrypts every blob to the current recipients without - changing the set (idempotent heal / backfill after a manual edit). Absence - of `recipients.txt` preserves exact legacy single-key behavior; the first - `recipients add` on a legacy store bootstraps the file seeded with the - local pubkey plus the new key. `init` now seeds `recipients.txt` born-multi - with the freshly generated pubkey. - -### Changed - -- **`secrets rekey` on a multi-recipient store** no longer generates a new - keypair — instead it re-encrypts all blobs to the current `recipients.txt` - set (the shared `_reencrypt_all` engine). On a legacy store (no - `recipients.txt`) `rekey` keeps today's generate-new-keypair behavior. -- **`secrets which`** now prints a `recipients: N (name, …)` line, or - `recipients: single-key (no recipients.txt)` for a legacy store. -- **`secrets verify` / `verify --all`** assert that each blob's age - recipient-stanza count equals the number of entries in `recipients.txt` - (skipped on legacy stores). Exits non-zero on any count mismatch so it can - gate CI or a migration. - -## [0.7.4.0] - 2026-06-18 - -### Added - -- **`secrets upgrade` verb (EGB-716)** — the fix path paired with the EGB-713 - version-skew *warning*. Until now the warning told you you were behind but not - how to catch up; `secrets upgrade` closes that loop. - - **`secrets upgrade`** — `git -C "$SCRIPT_DIR" pull --ff-only` on the tool's - own checkout (fast-forward only — never merges or rewrites local commits), - reports `vOLD -> vNEW`, then best-effort re-checks the store's recorded - writer-version against the new version so you see whether the EGB-713 nudge - is now cleared (the new code itself takes effect on your next command). - - **`secrets upgrade --check`** — reports whether an update is available - (`git fetch` + compare to upstream) and changes nothing. - - Deliberately thin: no auto-update, no background polling (this is a security - tool). Directed errors for not-a-git-checkout, no upstream, a diverged/dirty - branch, or being offline. - -## [0.7.3.1] - 2026-06-18 - -### Changed - -- **EGB-677 stage-1 structural cleanups (EGB-701)** — tech-debt dedup with one - new safety warning; no behavior change for the manifest-driven (v2) happy path. - - **`secrets which` now reuses the one external-entry extractor** the push/pull - path uses (`_json_external_entries`) instead of its own duplicated `jq` - projection. So `which` applies the same `properties`→`gradle-properties` - normalization and skips (with a warning) the same malformed external entries - the sync path drops — `which` shows exactly what will sync, not a stale raw - projection that could drift from the real behavior. - - **The two external-manifest read guards are factored into shared helpers** — - `_json_readable` (plain regular file, silent) and `_legacy_readable` (warns - and skips a symlinked `.secrets-files`) — so `_external_entries_for_push` and - `_external_entries_for_pull` can't drift apart. - -### Fixed - -- **Legacy (manifest-less) `pull` no longer silently under-restores (EGB-701)** — - the manifest-less pull path globs only top-level `*.age`/`.*.age`, so a nested - dotenv blob (`/.age`) written by a manifest-driven push on - another machine was invisible: restored nothing, counted nothing, said nothing. - It now **warns** and names each nested blob it can't reach (external blobs are - excluded — `pull_external_files` handles those), pointing at committing a - `.secrets.json` as the fix. The manifest-driven pull already restored nesting - correctly; this only closes the legacy path's blind spot. - -## [0.7.3.0] - 2026-06-08 - -### Added - -- **Real install / onboarding scripts (EGB-671)** — onboarding a machine is now - (close to) one command, and a mis-copied key fails loudly instead of silently. - - **`secrets join --remote --key `** — second-machine onboarding in - one verb: clones the vault, installs the key at mode 600, and **verifies the - key actually decrypts the store before declaring success**. An empty vault - reports "nothing to verify yet" (it never prints a false `VERIFIED`); a wrong - key fails loudly with the store left in place to fix. All security logic - (store resolution, URL handling, path rails) is reused from the audited core, - not re-implemented in a side script. - - **`secrets init --remote `** — wires the remote and pushes the initial - store so the upstream branch exists, so your first project `push` doesn't trip - the fast-forward-pull guard on a brand-new empty remote. Run interactively, - `init` also offers to add your first project's secrets (default No, skipped - under `--yes` / non-interactive, so it stays a clean primitive for CI). - - **`install.sh`** — thin bootstrap that ships in the repo: checks `age` + `jq` - + `git`, then prints the `PATH` line, the onboarding next-steps, the upgrade - one-liner, and a key-transfer hint. It never edits your shell config and never - runs `sudo` (it prints the command so you stay in control). - - **First-manifest `options.autoAdd` prompt (EGB-677 contract #2)** — the first - `push` that scaffolds a project's manifest now records an explicit, committed - `options.autoAdd` value (asked once when interactive; the default ON, written - explicitly, under automation). - -### Fixed - -- **Day-2 silent decrypt failure** — `secrets pull` now dies loudly when a blob - fails to decrypt with the current key (all three decrypt paths), instead of - emitting a warning and continuing with exit 0. A wrong key can no longer pass - unnoticed after onboarding. -- The `secrets init` second-machine trap now points at `secrets join` (the real - one-command path) instead of a manual `git clone`. - -## [0.7.2.0] - 2026-06-08 - -### Added - -- **`secrets list --json` (EGB-699)** — machine-readable listing for tooling and - CI. Emits a single JSON object on stdout: `{"store", "projects": [{"name", - "entries": [...]}]}`, where each entry self-describes via a `type` - discriminator — `{"type":"dotenv","path":}` or - `{"type":"external","subtype":"properties"|"file","path":}`. Reflects the - same recursive store walk as the human `list` (nested `/.age` - + `external/.age`). jq does the assembly so paths escape correctly; the - human store hint is suppressed so stdout stays pure JSON (notices → stderr). - jq is required only for `--json`. Feeds the EGB-671 install scripts, which need - to enumerate a cloned store programmatically instead of scraping the table. - -## [0.7.1.0] - 2026-06-08 - -### Added - -- **Version-skew nudge (EGB-713)** — the store now records the highest `secrets` - version that has written to it (`.secrets-writer-version`, committed, - monotonic). When you run a command against a store last written by a *newer* - `secrets` than your own, you get a one-line non-fatal stderr nudge to update - your tool; `secrets which` shows the store's `written-by:` version (and flags - when you're behind). Stores written by older builds carry no stamp and stay - silent — no false alarms. The loud counterpart to EGB-712's quiet - forcing function. - -## [0.7.0.0] - 2026-06-08 - -### Changed - -- **Additive store-format v2 (EGB-712)** — upgraded `secrets` clients now read - either external blob suffix (`.properties.age` or the legacy - `.gradle-properties.age`) and **dual-write** a `properties` external whenever a - v1 twin already exists in the store. Existing externals keep working for - teammates on an older `secrets`; only a brand-new `properties` external is - written v2-only (a gentle "upgrade to see it" forcing function). dotenv and - whole-`file` externals are unchanged across formats and always propagate. -- **`secrets migrate --finalize` is now optional GC**, not a required milestone. - Because clients dual-write and read-fall-back, no teammate is ever cut off by - *not* finalizing; finalize only reclaims the duplicate v1 blobs, and stays - deferrable indefinitely. Its safety gates are unchanged. This defuses the - cross-machine "all clients must be v2 before finalize" coordination gate. - -### Added - -- **`secrets migrate --status`** now reports `v2-only` externals per project - (the ones an un-upgraded client cannot read), so you can see the forcing - function's footprint at a glance. - -## [0.6.1.0] - 2026-06-08 - -### Changed - -- **`secrets migrate` copy-forward is now manifest-free (EGB-710)** — the - per-project step enumerates the store's `*.gradle-properties.age` blobs - directly (the same source of truth `--finalize` uses) instead of reading - `.secrets.json`. A legacy `.secrets-files`-only project now migrates cleanly - instead of dead-ending with "No .secrets.json", and a store blob the manifest - no longer declares still gets a v2 twin (so `--finalize` won't refuse it). - Running migrate in a project with no v1 properties blobs is a clean no-op. - -### Added - -- **`secrets migrate --status`** — a read-only survey that walks every project - in the store and reports its v2 readiness (v2-ready / migrated / NEEDS - MIGRATE), then whether the store as a whole is finalize-ready. Exits non-zero - while any v1 blob is un-twinned, so it can gate the path to `--finalize`. - -## [0.6.0.1] - 2026-06-08 - -### Added - -- **`secrets which` now prints the manifest version (EGB-700)** — the manifest - header line shows `version N` alongside the store format, so a single - `secrets which` tells you both the on-disk store format and the `.secrets.json` - schema version at a glance. - -## [0.6.0.0] - 2026-06-07 - -### Added - -- **Self-describing store format (v2) + `secrets migrate` (EGB-703)** — the - store now records its format in a committed `.secrets-format` file, and - `secrets which` prints it (`format: v2`). A fresh `secrets init` creates a - v2 store; existing stores read as v1 until migrated. -- **`secrets migrate`** — copy-forward a project's encrypted blobs to the v2 - layout. It is non-destructive: the old blobs are kept until you finalize, so - a half-migrated store stays fully readable and recoverable. `secrets migrate - --dry-run` previews exactly what would change without writing anything. -- **`secrets migrate --finalize`** — the one destructive step, run once - store-wide. It refuses unless `secrets verify` passes and every blob has its - new-format twin, cuts a `pre-v2-migrate-*` recovery tag first, then drops the - old blobs. It asks for confirmation (or `--yes`) because a machine still on - an older `secrets` will stop seeing migrated external files until it updates. - -### Changed - -- The external `properties` blob is stored as `.properties.age` in a v2 - store (was `.gradle-properties.age`), matching the manifest `type`. - `push`, `pull`, and `verify` pick the right name automatically from the store - format, so v1 and v2 stores both keep working during a migration. - -## [0.5.0.0] - 2026-06-07 - -### Added - -- **`secrets verify` (EGB-698)** — a read-only integrity check. Run it in a - project to cross-check the committed `.secrets.json` against the store both - ways (entries declared but missing from the store, and stored blobs with no - manifest entry) and decrypt-test every blob with your current key. Catches a - partially-synced store, a stale key, or a manifest that has drifted from the - store. Plaintext is streamed to `/dev/null` and never written to disk. -- **`secrets verify --all`** — decrypt-tests every blob in every project in the - store: a fast store-wide integrity sweep. (The store carries no manifests, so - `--all` checks decryptability only, not manifest consistency.) -- Both modes recurse the whole project tree, so nested entries and external - files are covered. `secrets verify` exits non-zero on any problem, so it can - gate CI or a future store migration. - -## [0.4.0.0] - 2026-06-07 - -### Added - -- **`.secrets.json` manifest (EGB-677 stage 1)** — a committed, project-root - manifest is now the source of truth for what syncs. List the env files you - want under `dotenv[]` (project-relative, nested paths and `@`-scoped - workspaces allowed; `..`, absolute, and symlink paths are rejected) and - out-of-project files under `external[]` (`properties` or `file`). The - manifest is shared across machines, so a teammate who clones the project - sees exactly what to pull. -- **`secrets add `** — declare an env file in the manifest without - pushing. Bootstraps `.secrets.json` on first use, dedupes, and writes a - stable canonical form. -- **Auto-add on push** — `secrets push` discovers new `.env*` / `.dev.vars` - files and adds them to the manifest (prints what it added and reminds you to - commit). Gated by `options.autoAdd` in the manifest (default on); - `push --frozen` syncs only declared files, and `push --dry-run` previews - what would change without writing anything. -- **Manifest-driven pull** — restores every declared file, recreating nested - directories as needed, with the same path-safety rail applied at restore - time so a malicious committed manifest can't write outside the project. An - empty manifest is a safe no-op. -- **Legacy `.secrets-files` absorb** — an existing `.secrets-files` is folded - into `.secrets.json` on first push (gradle-properties entries become - `properties`); on pull the legacy file is superseded with a warning. -- **Platform-aware install hints** — missing-dependency errors now print the - right install command for your platform (brew / apt-get / dnf). - -### Changed - -- `jq` is required only when a manifest is present or being written; - manifest-less projects keep working without `jq` (manifest features are - skipped with a notice). - -### Fixed - -- **Key rotation no longer orphans nested or external blobs.** `secrets rekey` - and `secrets list` now walk the entire project tree, so nested manifest - entries (`/.age`) and `external/` blobs are re-encrypted - and listed correctly. Previously a rekey could leave nested blobs encrypted - under the discarded old key, making them permanently undecryptable. -- Test assertions now fail correctly under system bash 3.2 (standalone - `[[ ]]` checks no longer pass silently). - -## [0.3.0.0] - 2026-06-07 - -### Added - -- **`file` external type (EGB-652)** — `.secrets-files` can now sync whole - files outside the project root (binary-safe; built for the Beacon Android - upload keystore): `file ~/keystores/beacon-upload.keystore`. Push encrypts - the file verbatim into `/external/`; pull restores it with mode - 600, backing up a divergent existing target to `.secrets-bak`. Same - path safety rails as `gradle-properties` (inside `$HOME`, no `..`, no - symlinks) minus the basename restriction. 7 new bats tests. - -## [0.2.1.0] - 2026-06-05 - -### Fixed - -- **`secrets rekey` no longer bricks dotenv stores.** The re-encrypt loop used a bare `"$dir"*` glob, which never matches dotfiles — so `.env` blobs were decrypted to the temp dir but never re-encrypted, leaving them on the **old** key while the new key overwrote `key.txt`. After a rotation, every dotenv file in the store was undecryptable. The glob now mirrors the decrypt loop (`"$dir"* "$dir".*`), and a round-trip test (push → rekey → pull) pins it. If you ran `rekey` on an earlier version and `pull` now fails with `no identity matched any of the recipients`, your blobs are on a pre-rotation key — recover with an old `key.txt` from another machine. -- **`secrets init` on a second machine now fails helpfully instead of half-initializing.** Copying `key.txt` into `~/.secrets` and then running `init` (instead of cloning your secrets repo) used to run `git init`, crash on the existing key, and leave a store with no `.gitignore` — a state where a later `push` would commit the private key. The guard now fires *before* `git init`, leaves the key untouched, and prints the exact `git clone` command to run — using the real remote URL when your `.secrets-store` file declares one. - -### Security - -- **The private key can no longer be committed by a store missing its `.gitignore`.** `push`, `pull`, and `rekey` now self-heal store protections immediately before any `git add -A`: a missing *or corrupted* `.gitignore` (one without the `key.txt` line) is rewritten, and the pre-commit hook is reinstalled if absent. The heal runs *after* the fast-forward pull, closing a window where remote history without a `.gitignore` could strip protection mid-push. -- **An already-tracked `key.txt` is now untracked automatically.** `.gitignore` can't untrack a file that was committed in the past; the heal now removes a tracked key from the index with a warning that history may need scrubbing and the key may warrant rotation. - -### Changed - -- Project `CLAUDE.md` gained agent skill-routing guidance and an updated test-suite count (126 bats tests, up from 113). - ## [0.2.0.0] - 2026-05-26 ### Added @@ -475,6 +86,6 @@ and this project adheres to a four-digit MAJOR.MINOR.PATCH.MICRO version scheme. - 37 → 66 tests. New coverage: store resolution rules and precedence, walk-up boundaries, command-injection prevention, key-file re-derivation across stores, teammate-onboarding error path, monorepo workspace binding, F1–F5 adversarial regressions. -[0.1.1.0]: https://git.dev.egbt.com/egbt/secrets/releases/tag/v0.1.1.0 -[0.1.0.1]: https://git.dev.egbt.com/egbt/secrets/releases/tag/v0.1.0.1 -[0.1.0.0]: https://git.dev.egbt.com/egbt/secrets/releases/tag/v0.1.0.0 +[0.1.1.0]: https://codeberg.org/egbt/secrets/releases/tag/v0.1.1.0 +[0.1.0.1]: https://codeberg.org/egbt/secrets/releases/tag/v0.1.0.1 +[0.1.0.0]: https://codeberg.org/egbt/secrets/releases/tag/v0.1.0.0 diff --git a/CLAUDE.md b/CLAUDE.md index a0261b8..e31f76a 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -16,91 +16,28 @@ cd ~/my-project && ./secrets pull # Pull + decrypt .env* files ```bash brew install bats-core -bats test/ # runs secrets.bats + manifest.bats + migrate.bats + upgrade.bats + recipients.bats -./test/run-security.sh # security regression subset + operator sign-off (see below) +bats test/secrets.bats ``` -**bash 3.2 assertion gotcha:** bats runs under system bash 3.2, where a -failing `[[ ]]` mid-test does NOT fail the test (the ERR trap skips `[[` -compound commands). Every standalone `[[ ... ]]` assertion MUST end with -`|| false`. Single-bracket `[ ]` assertions are unaffected. - -## Security review policy (mandatory for agents) - -This repo contains **defensive** security regression tests with deliberate -attack-payload fixtures (shell injection strings, symlink escape cases, path -rails). Hosted models (including Opus) block AI-driven "think like an attacker" -/ red-team / security-specialist review of that material under Anthropic Usage -Policy (real-time cyber safeguards). - -**Read `.ship-policy.json` before any `/ship`, `/review`, or `/autoplan` work.** - -When that file is present (it always is in this repo): - -1. **Never** dispatch red-team subagents, security specialist subagents, or - adversarial review prompts ("think like an attacker", "find security holes", - "chaos engineer", exploit hunting) against this codebase. -2. **Never** run `./test/run-security.sh` or filter bats to SECURITY/adversarial - tests on behalf of the user — that is operator-local work only. -3. **Before ship/PR:** ask the human operator to run `./test/run-security.sh` - locally and complete the SIGNOFF prompt. Record lives at - `.gstack/security-signoff.json` (gitignored). Proceed with ship only after - they confirm sign-off for the current `HEAD` (or note in the PR that sign-off - is pending). -4. Normal `./ship` steps may continue (tests via `bats test/`, eng review, - changelog, PR) — only the AI security/red-team/adversarial passes are opted out. - -Gstack `/ship` reads the same policy via `gstack-ship-policy` (sources -`.ship-policy.json` at repo root): Step 9 blocks when manual sign-off is missing, -skips security specialist + red team, and Step 11 skips adversarial review. - ## Architecture -Single bash script (`secrets`) with subcommands: init, push, pull, list, rm, rekey, verify, migrate, recipients, reencrypt, sync, upgrade. +Single bash script (`secrets`) with subcommands: init, push, pull, list, rm, rekey. - Encryption: `age` with key files (not passphrases — age passphrases are non-scriptable) - Storage: Private git repo at `~/.secrets/` - Convention: Tracks `.env`, `.env.*`, and `.dev.vars` (not `.envrc`, `.environment-*`) -- Manifest (EGB-677 stage 1): committed `.secrets.json` is the source of truth for what syncs — `dotenv[]` (project-relative, nested ok, `@` allowed; rail rejects `..`/absolute/symlink) + `external[]` (`properties`/`file`). Push discovery auto-adds (gated by committed `options.autoAdd`, default ON; `--frozen`/`--dry-run` overrides), bootstraps the manifest on first push (written only after ≥1 blob encrypts), and absorbs a legacy `.secrets-files` (gradle-properties → `properties`; on pull the legacy file is superseded with a warning). Store layout: nested dotenv entries land at `/.age` (relpath preserved — the store self-describes where a file restores). jq is a hard dep only when a manifest exists/is written; manifest-less projects run jq-free (manifest features skipped with a notice). `check_cmd` prints platform-aware install hints. -- Store format (EGB-677 stage 2 / EGB-703): the store is self-describing via a committed one-line `$SECRETS_DIR/.secrets-format` file (`2`). Absence ⇒ v1 (every store predating EGB-703). v2's only on-disk change vs v1 is the external `properties` blob suffix: `.gradle-properties.age` → `.properties.age` (matching the manifest `type`); dotenv and `file` blobs are unchanged. `_store_format()` reads the marker. **Additive v2 (EGB-712):** reads resolve a `properties` blob by trying `.properties.age` then falling back to `.gradle-properties.age` (`_resolve_external_blob_read`); writes dual-write a `properties` external only when a v1 twin already exists in the store (`_external_blob_write_targets`), so existing externals keep old clients fresh while brand-new externals are written v2-only (a gentle forcing function). Blob location no longer depends on the marker — the old `_external_blob_suffix` is gone. `init` stamps a fresh store v2 (born-v2). `secrets which` prints the store-format line `format: vN`, and (EGB-700) when a `.secrets.json` is present the manifest header line also carries its schema version (`manifest (.secrets.json at , version N):`). **Migration is copy-forward and non-destructive:** `secrets migrate --dry-run` (per project, reports old→new, writes nothing) → `secrets migrate` (per project, manifest-free: enumerates the store's `*.gradle-properties.age` blobs directly — same source of truth as `--finalize` — and writes their `.properties.age` twins, so a legacy `.secrets-files`-only project with no `.secrets.json` migrates cleanly and no store blob is left un-twinned; idempotent; EGB-710) → `secrets migrate --finalize` (store-wide; the ONLY destructive step — gates on `verify --all` green + every v1 blob having a v2 twin, cuts a `pre-v2-migrate-` recovery tag, stamps the marker, then drops v1 blobs; refuses without `--yes`/operator confirmation since a lagging v1 client against a finalized store stops seeing `properties` externals until it upgrades). `secrets migrate --status` is a read-only survey that walks every project in the store and reports each one's v2 readiness (v2-ready / migrated / NEEDS MIGRATE, plus a `v2-only` count of externals old clients can't read), exiting non-zero while any v1 blob is un-twinned so it gates the path to `--finalize` (EGB-710/EGB-712). **Under additive v2 (EGB-712) `--finalize` is now OPTIONAL GC, not a required milestone:** because upgraded clients dual-write existing externals and read-fall-back, *not* finalizing never cuts anyone off — finalize only reclaims the duplicate v1 blobs and stays deferrable indefinitely (defusing the cross-machine coordination gate). dotenv and `file` blobs are identical across formats, so they always propagate to old clients; only a brand-new `properties` external is v2-only. **Version-skew nudge (EGB-713):** a committed `$SECRETS_DIR/.secrets-writer-version` records the highest client `VERSION` that has written to the store (monotonic; stamped via `_stamp_writer_version` right before each store-committing `git add -A` — push/rekey/migrate/finalize — never on read paths, so it always rides a commit and never dangles to break `pull --ff-only`). `check_initialized` calls `_check_store_version_skew`, which warns once per invocation (stderr, non-fatal, `set -e`-safe) when the store's stamp is numerically greater than `_client_version` (read from `$SCRIPT_DIR/VERSION`); `secrets which` prints the `written-by:` line. Stores with no stamp (pre-EGB-713) are silent. The deliberate flatten-to-basename naming the EGB-677 CEO plan sketched was dropped as lossy (it discards the restore relpath that makes the store self-describing) — see the EGB-703 eureka. **Upgrade verb (EGB-716):** `secrets upgrade` is the fix path paired with the EGB-713 skew *warning* — it `git -C "$SCRIPT_DIR" pull --ff-only`s the tool's own checkout (fast-forward only, never merges/rewrites local commits), reports `vOLD -> vNEW`, then best-effort re-checks `_store_writer_version` against the new on-disk version so the operator sees whether the nudge is cleared (the new code takes effect next invocation). `secrets upgrade --check` does `git fetch` + `rev-list --count HEAD..@{u}` and reports availability without pulling. Deliberately thin: no auto-update, no background polling (security tool). Directed errors for not-a-git-checkout / no-upstream / diverged / offline. `cmd_upgrade` never calls `check_initialized` (it's about the tool, not the store); the skew re-check is silent unless a store with a writer-version resolves. -- Store sync + divergence (EGB-1230/EGB-1231): the store is a git repo, so a clone can end up ahead of and behind its remote at once. **EGB-1230:** `cmd_pull`'s sync used to be `git pull >/dev/null 2>&1` under `set -euo pipefail` — a store that couldn't fast-forward killed the script there with git's exit 128 and nothing on either stream (a banner, no files, no reason; invisible in a pipeline). It now routes through `_store_sync_pull`, which guards the pull, captures git's output as the diagnosis, and dies naming the store path and `secrets sync`. That sync is **`--ff-only`**, matching the push path — a plain `git pull` could quietly manufacture a merge commit in the store, and divergence is now resolved in exactly one place. **EGB-1231:** `_store_git_state` emits `ahead\tbehind\tdirty` (from `rev-list --left-right --count @{u}...HEAD` plus `status --porcelain`) and `_format_store_state` renders it; `cmd_which` prints a `remote:` line from them — offline-safe (reports against the last fetch), silent with no remote/upstream. `cmd_sync` is the reconcile verb the CLI was missing: fetch → report state → stash (`push -u`) → `rebase @{u}` → restore stash → `ensure_store_protections` (rebased-in history may lack `.gitignore`, and a store missing the `key.txt` line would stage the private key — same reasoning as push) → **confirmation-gated** `git push` of local commits. The gate (`_sync_confirm_push`) reads `/dev/tty` and requires a tty, so it stays CLOSED in scripts/CI rather than publishing to a shared store by default; `--yes` opens it, `--dry-run` reports and returns before any mutation. Non-destructive by construction: no merge, no `--force`, no `reset --hard`, no `stash drop`. A rebase conflict collects the conflicting paths BEFORE `rebase --abort` (the abort clears them), restores the stash, and dies — store byte-identical to how it was found. `_sync_restore_stash` never drops the stash on a failed pop; it tells the operator where their only copy lives. `cmd_sync` does not `_stamp_writer_version`: it replays existing commits rather than authoring content, and the stamp is specified to ride a store-committing `git add -A`. Test suite: `test/sync.bats` (25 tests), including a grep over the `cmd_sync` body asserting the destructive git verbs never appear in it. -- Verify (EGB-698): `secrets verify` is a read-only integrity check. Default mode (current project) cross-checks `$PWD/.secrets.json` against `$SECRETS_DIR//` both ways (declared-but-missing blobs + orphaned blobs) and decrypt-tests every blob (dotenv + external) by streaming plaintext to `/dev/null` (never written to disk). `secrets verify --all` decrypt-tests every blob in every project (integrity only — the store carries no manifests, so consistency can't be checked store-wide). Both recurse the whole project tree (`find -type f`, same as rekey/list). Exits non-zero on any finding so it can gate the stage-2 `migrate --finalize` and CI. The store deliberately holds no manifest — `.secrets.json` is committed in each project's own repo and read from `$PWD`. -- External files: `.secrets-files` manifest tracks designated keys from files outside the project (e.g. `~/.gradle/gradle.properties`, merged not overwritten — EGB-531) and whole binary files (type `file`, e.g. an Android upload keystore — EGB-652); see below -- Workspaces (EGB-1232): `--workspaces` and the plain-push workspace re-scan both resolve patterns through ONE source — `_workspace_patterns()`. It reads `package.json` `.workspaces` via `$WORKSPACES_JQ` (type-aware: handles npm's array AND yarn's object `{packages:[...]}` form) and falls back to `pnpm-workspace.yaml`'s `packages:` block when package.json declares none. **Two defects fixed:** (1) both call sites were package.json-only, so no pnpm monorepo ever resolved a workspace — and `_maybe_workspace_env_files` failed *silently* (`jq -e '.workspaces' ... || return 0`), making auto-discovery inert and `push` print "Nothing new to add", indistinguishable from a repo with nothing new; it bit the same repo twice. (2) the old filter `.workspaces // .workspaces.packages | .[]` short-circuits on yarn's truthy object, iterating the object's values and yielding the pattern ARRAY as a single token. Note a naive reorder does NOT fix it — `.workspaces.packages` errors on an array; hence the `if type == "object"` form. `_pnpm_workspace_packages()` is a deliberate non-parser (block sequence only, stops at the next top-level key so pnpm 10's `onlyBuiltDependencies:`/`catalog:` can't leak in as globs, strips quotes/inline comments, refuses a symlinked file). Patterns are validated by `_valid_workspace_pattern` before they reach the unquoted `for pattern in $patterns` glob expansion (no absolute/`..`/metacharacters/whitespace; pnpm `!` negations skipped) — same posture as `.secrets-store`/`.secrets-files`. `_looks_like_monorepo` + `_workspace_source` turn the old silent return into a warning that names the real file, and `get_workspaces`'s error names `pnpm-workspace.yaml` when that's what's present instead of blaming package.json. jq is required only when package.json is the source. **Scope note:** the workspace re-scan still runs only for projects that already have a `.secrets.json` — push's root-scan-only behavior on a first push is by design (EGB-677 E13), and EGB-1232 is about the fallback that covers it never engaging. Tests: `test/workspaces.bats` (18). +- External files: `.secrets-files` manifest tracks designated keys from files outside the project (e.g. `~/.gradle/gradle.properties`) — merged, not overwritten (EGB-531, see below) +- Workspaces: `--workspaces` flag reads `package.json` workspaces, requires `jq` - Safety: Pre-commit hook rejects plaintext secret files (`.env`, `.dev.vars`, `gradle.properties`) -- Multi-recipient (EGB-283): a store-scoped, committed `recipients.txt` (age `-R` - format, `# name` comments) lets one store encrypt every blob to N age keys — - one per team member. Managed via `secrets recipients add/rm/list`; absence of - the file ⇒ legacy single-key behavior (recipients = the pubkey derived from - `key.txt`). The file is parsed by us (never `age -R `) into a validated - `RECIPIENT_ARGS` array (native age X25519 only, `age1[0-9a-z]{58}`; SSH - recipients rejected; symlinked file refused) — same conservative posture as - `.secrets-store`/`.secrets-files`. `_load_recipients` populates the array; - every encrypt site routes through it. Any recipient change re-encrypts the - WHOLE store in one commit via the shared `_reencrypt_all` engine (also used by - the new `secrets reencrypt` and by `rekey` on a multi-recipient store, where - rekey re-encrypts to the set with NO new keypair; legacy stores keep rekey's - generate-new-keypair behavior). `init` seeds `recipients.txt` born-multi. - `which` prints `recipients: N`; `verify`/`verify --all` assert each blob's - age recipient-stanza count equals `recipients.txt`'s length. Removal takes - effect going forward — git history stays readable by an old key, so rotate - genuinely-sensitive values. Decryption is unchanged (each member uses their - own `key.txt`). - Portability: must run on system bash 3.2 (macOS) — no associative arrays or bash-4 features ## Project Structure ``` -secrets # CLI script (~2000 lines bash) +secrets # CLI script (~600 lines bash) hooks/pre-commit # Pre-commit hook template test/ - secrets.bats # bats-core test suite (140 tests) - manifest.bats # EGB-677 .secrets.json manifest tests (83 tests) - migrate.bats # EGB-703 store-format-v2 migration tests (35 tests) - upgrade.bats # EGB-716 `secrets upgrade` self-update tests (8 tests) - sync.bats # EGB-1230/1231 store sync + divergence reconcile tests (25 tests) - workspaces.bats # EGB-1232 npm/yarn/pnpm workspace discovery tests (18 tests) - recipients.bats # EGB-283 multi-recipient age encryption tests (34 tests) + secrets.bats # bats-core test suite (113 tests) test_helper.bash # Shared setup/teardown README.md # User-facing documentation CLAUDE.md # This file @@ -125,12 +62,12 @@ The active store directory is picked by `resolve_store()` using these rules, hig ## External files (.secrets-files) — EGB-531 -`.secrets-files` is a committed, project-root manifest declaring keys to sync from files **outside** the project (motivating case: `~/.gradle/gradle.properties`, which Android Studio GUI builds read but terminal env vars can't reach). One entry per line: ` ...`. Two types: `gradle-properties` (named-key merge) and `file` (EGB-652 — whole-file verbatim sync, binary-safe, built for the Beacon Android upload keystore; no keys, restored at mode 600 with a `.secrets-bak` backup of a divergent existing target, basename restriction waived but all other path rails apply). Still no plugin-dispatch framework — each type is a concrete `case` branch (deliberate scope cut). +`.secrets-files` is a committed, project-root manifest declaring keys to sync from files **outside** the project (motivating case: `~/.gradle/gradle.properties`, which Android Studio GUI builds read but terminal env vars can't reach). One entry per line: ` ...`. Only type `gradle-properties` is supported; the type token leaves room for future types **without** a plugin-dispatch framework (build the concrete case — a deliberate scope cut). Key design decisions (all driven by /autoplan review): - **Wire-in is at command scope** (`cmd_push`/`cmd_pull`), via `push_external_files` / `pull_external_files`, **not** inside `push_dir_to_project` / `pull_project_to_dir` (those loop per-workspace and `pull_project_to_dir` uses stdout as a data channel). -- **Storage:** blobs live in `$SECRETS_DIR//external/.gradle-properties.age`. The `external/` subdir keeps them out of the legacy non-recursive `*.age` / `.*.age` globs the dotenv `pull` path uses, so a dotenv pull can never decrypt an external blob into cwd. `cmd_rekey` and `cmd_list` instead walk the **entire** project tree (`find -type f`), so they cover both `external/.age` and nested manifest dotenv blobs (`/.age`) — rekey MUST recurse, or any nested/external blob is orphaned under the old key after rotation = data loss (EGB-677 regression test: "rekey re-encrypts a nested manifest dotenv blob"). **EGB-701 cleanups:** (1) the *legacy* (manifest-less) `pull` keeps its non-recursive globs but now **warns** when nested `/.age` blobs exist that those globs can't see (it excludes `external/`, which `pull_external_files` handles) — so a manifest-less pull never silently under-restores; the fix the warning points at is committing a `.secrets.json`. (2) `cmd_which`, push, and pull share one external extractor (`_json_external_entries`), so `which` applies the same `properties`→`gradle-properties` normalization and skip-with-warning rules the sync path does (it shows exactly what will sync, not a stale raw projection). (3) the two external-manifest read guards are factored into `_json_readable` (plain regular file, silent) / `_legacy_readable` (warn-and-skip on a symlinked legacy manifest). `` = manifest path token with non-`[A-Za-z0-9._-]` chars → `_`, plus a `cksum` suffix of the original path so paths that clean to the same string (`a/b` vs `a_b`) don't collide. Machine-independent (derived from the committed manifest token, not the expanded path). `cmd_list --json` (EGB-699) emits the same recursive walk as a machine-readable object (`{store, projects[].entries[]}`, each entry `dotenv`→`path` or `external`→`subtype`+`path`) for tooling/CI (feeds EGB-671); jq assembles it so paths escape correctly and stdout stays pure JSON (the human store hint is suppressed; jq is a hard dep only in `--json` mode). +- **Storage:** blobs live in `$SECRETS_DIR//external/.gradle-properties.age`. The `external/` subdir keeps them out of the existing non-recursive `*.age` / `.*.age` globs (pull, list, rekey), so the old dotenv path can never decrypt a blob into cwd. `cmd_rekey` and `cmd_list` recurse into `external/` explicitly (rekey MUST, or the blob is orphaned after rotation = data loss). `` = manifest path token with non-`[A-Za-z0-9._-]` chars → `_`, plus a `cksum` suffix of the original path so paths that clean to the same string (`a/b` vs `a_b`) don't collide. Machine-independent (derived from the committed manifest token, not the expanded path). - **Merge is pure bash, no `sed`/regex** (`merge_gradle_keys`): exact-string key comparison (avoids `beaconClerkPk` vs `beaconClerkPkTest` substring bug), value treated as opaque literal (survives `& \ /` in values). Updates a managed key in place at its first occurrence, collapses duplicates, appends new keys, preserves unrelated lines/comments/order. Continuation lines (trailing odd backslashes, tracked by `_trailing_bs_odd`) are never matched as keys. Atomic write: temp in the same dir → `chmod` to match (or `600` on create) → `mv`. Backs up to `.secrets-bak` before each merge. - **Properties separator parsing** (`_props_get`): key ends at the first `=`, `:`, or whitespace (after lstrip); handles `key=value`, `key = value`, `key:value`, `key value`; last definition wins. - **Security:** the write target comes from a committed file, so `_validate_external_target_path` locks it down — basename must be `gradle.properties`, must resolve inside `$HOME` (deepest-existing-ancestor resolved, symlink target/parent refused, `..` rejected). This blocks a malicious manifest from appending decrypted keys to `~/.gitconfig`/`~/.bashrc`. `_parse_secrets_files_manifest` rejects shell metacharacters/control chars in path and keys (path allows `[A-Za-z0-9/._~-]` only; keys allow `[A-Za-z0-9._-]` + space), mirrors the `.secrets-store` posture (no shell expansion, symlinked manifest skipped). @@ -138,63 +75,13 @@ Key design decisions (all driven by /autoplan review): ## Deploy Configuration -- Platform: NONE (distributed via `git clone` from the private Forgejo at `git.dev.egbt.com`) +- Platform: NONE (distributed via `git clone` from GitHub) - Production URL: N/A (no live service) - Release model: merge to `main` is the release. Optionally tagged with `v`. - Verification after merge: a fresh `git clone` should produce a working `secrets which` against an isolated `$HOME`. No canary URL. - Staging: none. - Rollback: revert the merge commit on `main` (and delete the tag) to roll back. -## Forge operations (self-hosted Forgejo) - -The remote is a private Forgejo instance at `https://git.dev.egbt.com` -(migrated off Codeberg 2026-09-08). `gh`/`glab` do NOT work here. Use `tea` -(login name: `egbt`, user `brian`) for forge operations when a skill's -platform detection comes up "unknown": - -**Always pass `--login egbt --repo egbt/secrets` explicitly.** `tea`'s repo -autodetection fails here ("remote repository required"), and this machine also -has a leftover `codeberg` login pointing at the *old* forge -(`https://codeberg.org`) that `tea` will silently fall back to in -non-interactive mode — which would target the wrong server. Confirm with -`tea logins list` if a command errors. - -- PRs: `tea pr create --login egbt --repo egbt/secrets --base main --head --title ... --description ...` / `tea pr merge --login egbt --repo egbt/secrets` -- Releases: `tea releases create --login egbt --repo egbt/secrets --tag v --title "v — " --note ...` - (convention: one release per tag, title `v — `) -- Issues/status: `tea issues --login egbt --repo egbt/secrets`, `tea pr list --login egbt --repo egbt/secrets` -- **SSH is on port 2222**, not 22 (port 22 is the host's own sshd). Clone/remote - URLs must be `ssh://git@git.dev.egbt.com:2222/egbt/secrets.git`. A bare - `git@git.dev.egbt.com:egbt/secrets.git` will fail with "Permission denied - (publickey)" because it hits the wrong daemon. -- The host resolves to a Tailscale address — the forge is reachable only on the - VPN. Off-net, push/pull/`tea` all fail to connect; that is expected, not a - credentials problem. -- `FORGEJO_URL` and `FORGEJO_TOKEN` (API token for user `brian`) live in - `~/.zshenv` for direct API calls. -- CI: the instance has an Actions runner available, but no workflow is - configured for this repo yet. The bats suite run locally is still the merge - gate. - ## Environment variable `SECRETS_DIR` overrides the default `~/.secrets` location (useful for testing). Per-project bindings via `.secrets-store` file beat this env var; use `--store ` for one-shot overrides that beat everything. - -## Skill routing - -When the user's request matches an available skill, invoke it via the Skill tool. When in doubt, invoke the skill. - -Key routing rules: -- Product ideas/brainstorming → invoke /office-hours -- Strategy/scope → invoke /plan-ceo-review -- Architecture → invoke /plan-eng-review -- Design system/plan review → invoke /design-consultation or /plan-design-review -- Full review pipeline → invoke /autoplan -- Bugs/errors → invoke /investigate -- QA/testing site behavior → invoke /qa or /qa-only -- Code review/diff check → invoke /review -- Visual polish → invoke /design-review -- Ship/deploy/PR → invoke /ship or /land-and-deploy (after reading `.ship-policy.json`; no AI adversarial/red-team/security-specialist review in this repo) -- Save progress → invoke /context-save -- Resume context → invoke /context-restore -- Author a backlog-ready spec/issue → invoke /spec diff --git a/README.md b/README.md index acfd2b4..2e23b41 100644 --- a/README.md +++ b/README.md @@ -54,101 +54,87 @@ flowchart TD Files like `.envrc` (direnv) and `.environment-*` are intentionally **not** tracked. -Beyond project files, `secrets` can also sync files that live *outside* the project — designated keys from `~/.gradle/gradle.properties` (merged without clobbering unrelated keys), or whole files like an Android upload keystore. See [External files (Gradle properties)](#external-files-gradle-properties). +Beyond project files, `secrets` can also sync designated keys from files that live *outside* the project — like `~/.gradle/gradle.properties` — merging them in without clobbering unrelated keys. See [External files (Gradle properties)](#external-files-gradle-properties). ## Prerequisites -- **macOS or Linux** -- **git** (`git --version` to check) -- **age** and **jq** — `install.sh` checks for these and prints the exact install command for your platform (Homebrew on macOS, `apt`/`dnf` on Linux) +- **macOS** (uses Homebrew for installation) +- **git** (already installed on most Macs — type `git --version` to check) +- **age** (the encryption tool — installed in step 1 below) ## Setup -Clone the tool repo, then run `install.sh`. It checks dependencies and prints the -two commands to finish setup. It never edits your shell config and never runs -sudo — it prints the commands so you stay in control. +### First machine (one-time setup) ```bash -git clone https://git.dev.egbt.com/egbt/secrets.git ~/dev/secrets -cd ~/dev/secrets -./install.sh +# 1. Install the encryption tool +brew install age + +# 2. Download the secrets tool (this repo — contains only the CLI, no secret files) +git clone https://codeberg.org/egbt/secrets.git ~/dev/secrets + +# 3. Make the 'secrets' command available everywhere +# Add this line to your shell config file (~/.zshrc on Mac): +export PATH="$HOME/dev/secrets:$PATH" +# Then restart your terminal, or run: +source ~/.zshrc + +# 4. Initialize your encrypted secrets store +# This creates a folder at ~/.secrets/ with your encryption key +secrets init + +# 5. Create a PRIVATE repository on GitHub to store your encrypted secrets +# Go to github.com/new, name it something like 'my-secrets', and make sure +# "Private" is selected. Then connect it: +cd ~/.secrets +git remote add origin git@github.com:/my-secrets.git +git push -u origin main ``` -`install.sh` prints a `export PATH="$HOME/dev/secrets:$PATH"` line — add it to your -shell config (`~/.zshrc` or `~/.bashrc`) and restart your terminal. Then onboard -this machine with one of the two flows below. +> **Important:** Step 5 creates a *separate* private repo for your encrypted secrets. This is different from the `secrets` tool repo you cloned in step 2. The tool repo can be public — it contains no secrets. The `~/.secrets/` repo must be private. -### First machine (new vault) +### Additional machines + +On each new machine (your desktop, a teammate's laptop, etc.): ```bash -# 1. Create a PRIVATE repo for your encrypted secrets (github.com/new or a -# GitLab/Forgejo private repo). It holds only ciphertext — never your key. -# Then wire it up and push the store in one command: -secrets init --remote git@github.com:/my-secrets.git +# 1. Install prerequisites and the tool (same as steps 1-3 above) +brew install age +git clone https://codeberg.org/egbt/secrets.git ~/dev/secrets +export PATH="$HOME/dev/secrets:$PATH" # add to ~/.zshrc -# 2. (optional) Add a project's secrets. From a project directory: +# 2. Clone the encrypted secrets repo +git clone git@github.com:/my-secrets.git ~/.secrets + +# 3. Copy the encryption key from your first machine +# This is the only step that requires direct machine-to-machine transfer. +# Choose one method: +# +# Option A: AirDrop (Mac to Mac) +# On your first machine, right-click ~/.secrets/key.txt → Share → AirDrop +# Save it to ~/.secrets/key.txt on the new machine +# +# Option B: Secure copy over SSH +# scp first-machine:~/.secrets/key.txt ~/.secrets/key.txt +# +# Option C: USB drive +# Copy key.txt to a USB drive, transfer it, delete from USB after + +# 4. Pull your secrets into any project cd ~/myapp -secrets push -# The first push asks once whether to auto-track new env files and records -# your choice in the project's .secrets.json. +secrets pull ``` -`secrets init --remote` generates your key (`~/.secrets/key.txt`), wires the -remote, and pushes the initial store so the upstream branch exists. The private -secrets repo is separate from this tool repo — the tool repo is public and holds -no secrets; the `~/.secrets/` repo must be private. - -> Running `secrets init` interactively (in a terminal) also offers to add your -> first project's secrets right away. Run it with `--yes` (or in any non-tty -> context like CI) to skip that prompt and just create the vault. - -### Other machines (join an existing vault) - -On a second machine, a desktop, or a teammate's laptop: - -```bash -# 1. Clone the tool and run the bootstrap (as in Setup above) -git clone https://git.dev.egbt.com/egbt/secrets.git ~/dev/secrets -cd ~/dev/secrets && ./install.sh # add the printed PATH line to your shell config - -# 2. Get key.txt onto this machine (the one manual, out-of-band step): -# AirDrop (Mac→Mac), or -# scp first-machine:~/.secrets/key.txt ~/Downloads/key.txt, or -# a USB drive (delete from the drive afterward) - -# 3. Join the vault in one command: -secrets join --remote git@github.com:/my-secrets.git --key ~/Downloads/key.txt -``` - -`secrets join` clones the vault, installs the key at mode 600, and **verifies the -key actually decrypts the store before declaring success** — a mis-copied key -fails loudly here, not silently on a later `secrets pull`. On success it tells you -to run `secrets pull` in any project. - > **The key file (`~/.secrets/key.txt`) is the only thing that needs to be transferred manually.** It never leaves your machines — it's excluded from git, never uploaded, never transmitted over the internet. Anyone with this file can decrypt all your secrets, so treat it like a password. ### Sharing with teammates -**Simple approach (shared key):** To share secrets with a teammate, they need: +To share secrets with a teammate, they need: -1. Access to your private secrets repo (add them as a collaborator) -2. A copy of `key.txt` (send it directly — AirDrop, USB, or in-person) +1. Access to your private `my-secrets` GitHub repo (add them as a collaborator) +2. A copy of `key.txt` (send it to them directly — AirDrop, USB, or in-person) -Everyone on the team uses the same key. A teammate joins with -`secrets join --remote --key `. When anyone runs -`secrets push`, the encrypted files update and everyone else runs `secrets pull` -to get the latest. - -### Updating the tool - -```bash -git -C ~/dev/secrets pull -``` - -If your store was last written by a newer client than yours, `secrets` prints a -one-line version-skew nudge — that's your cue to run the command above. - -**Per-teammate keys (recommended for teams):** Use `secrets recipients add` so each person keeps their own private key — no key sharing needed. See [Onboarding and offboarding teammates](#onboarding-and-offboarding-teammates) below. +Everyone on the team uses the same key. When anyone runs `secrets push`, the encrypted files are updated and everyone else can `secrets pull` to get the latest version. ## Usage @@ -174,71 +160,12 @@ secrets clear |---------|-------------| | `secrets init` | Create the `~/.secrets/` repo and generate an encryption key | | `secrets push` | Encrypt secret files in the current directory and upload them | -| `secrets push --frozen` | Sync only what `.secrets.json` declares (skip auto-add) | -| `secrets push --dry-run` | Show what would be added/synced without changing anything | | `secrets pull` | Download and decrypt secret files into the current directory | -| `secrets add ` | Declare a project-relative file in `.secrets.json` | | `secrets clear` | Delete plaintext secret files from the current directory | | `secrets run ` | Pull secrets, run a command, then clear secrets when it exits | | `secrets list` | Show all projects that have stored secrets | -| `secrets list --json` | Same listing as a machine-readable JSON object (`{store, projects[].entries[]}`, each entry `dotenv`/`external`) for tooling and CI. JSON goes to stdout; notices to stderr | | `secrets rm ` | Delete a project's secrets from the store | -| `secrets rekey` | Generate a new encryption key and re-encrypt everything (single-key store) or re-encrypt to the current recipients without changing keys (multi-recipient store) | -| `secrets verify [project]` | Check the current project's `.secrets.json` against the store (missing/orphaned blobs) and decrypt every blob. `[project]` overrides the store directory name; the manifest is still read from the current directory | -| `secrets verify --all` | Decrypt-test every blob in every project — a store-wide integrity sweep | -| `secrets migrate [--dry-run]` | Copy-forward this project's encrypted blobs to store format v2 (non-destructive; manifest-free; `--dry-run` previews) | -| `secrets migrate --status` | Survey every project's v2 readiness; exits non-zero until the whole store is finalize-ready | -| `secrets migrate --finalize` | **Optional GC** — drop the old v1 blobs and mark the store pure v2. Never required: upgraded clients dual-write and read-fall-back, so not finalizing never cuts anyone off | -| `secrets recipients list` | List the store's recipient public keys (and names if set) | -| `secrets recipients add [--name N]` | Add a recipient key to the store and immediately re-encrypt every blob to the new set | -| `secrets recipients rm [--yes]` | Remove a recipient and re-encrypt the store; `--yes` required when removing your own key | -| `secrets reencrypt` | Re-encrypt every blob to the current recipients (idempotent — useful after a manual edit or partial failure) | -| `secrets sync` | Reconcile a store that has diverged from its remote: stash local blob edits, rebase onto the remote, restore the stash, then offer to publish your local commits. Never merges, force-pushes, or hard-resets | -| `secrets sync --dry-run` | Report the store's ahead/behind/dirty state and what a reconcile would do; changes nothing | -| `secrets sync --yes` | Reconcile and publish local commits without the confirmation prompt (for scripts) | -| `secrets upgrade` | Self-update the tool: `git pull --ff-only` on the `secrets` checkout, report old → new version, then re-check store version-skew. No auto-update, no background checks | -| `secrets upgrade --check` | Report whether an update is available (without pulling); changes nothing | - -### Upgrading: do teammates on an older `secrets` get new secrets? - -Store-format v2 is **additive** — an upgraded client reads either blob suffix and keeps the old (v1) suffix alive for externals that already existed, so you almost never have to coordinate an upgrade: - -| Secret type | Old client gets it? | -|---|---| -| `.env` / `.env.*` / `.dev.vars` | **Yes, always** (blob name is identical across formats) | -| whole-file external (`file`) | **Yes, always** | -| `properties` external that already existed | **Yes** (dual-written so old clients stay fresh) | -| brand-new `properties` external | **No — must upgrade `secrets`** (the gentle forcing function) | - -"Upgrade your secrets" = `git pull` the tool clone (binary ≥ 0.6.0.0) and/or `secrets migrate` the store. A read-only teammate only needs the tool `git pull`. - -And you'll be told when you're behind: if a store was last written by a newer `secrets` than the one you're running, any command prints a one-line nudge to stderr (non-fatal) — and `secrets which` shows the store's `written-by:` version. Stores written by older builds (no version stamp) stay silent. - -### When the store diverges - -The store is a git repo, so two machines pushing at once can leave your clone -both ahead and behind its remote. `secrets push` needs a fast-forward and -`secrets pull` won't silently merge, so both stop and tell you to run: - -```bash -secrets sync -``` - -`sync` fetches, stashes any uncommitted blob edits, rebases your local commits -onto the remote, restores the stash, and then *asks* before publishing your -commits to the shared store (`--yes` skips the prompt; `--dry-run` just -reports). If the rebase conflicts, it aborts, restores your stash, names the -conflicting files, and leaves the store exactly as it found it — nothing in -the path force-pushes, hard-resets, or drops a stash. - -`secrets which` now reports the same state up front, so you can see it coming: - -``` -store: /Users/you/.secrets -source: default -format: v2 -remote: ahead 1, behind 11, 3 modified (run: secrets sync) -``` +| `secrets rekey` | Generate a new encryption key and re-encrypt everything | ### Automatic project detection @@ -249,32 +176,6 @@ When you run `secrets push` or `secrets pull` without specifying a project name, You can also specify a name explicitly: `secrets push myapp`. -### The manifest (`.secrets.json`) - -Every project gets a committed `.secrets.json` at its root declaring exactly what syncs — the single source of truth `push` and `pull` operate from (requires `jq`): - -```json -{ - "version": 2, - "options": { "autoAdd": true }, - "dotenv": [".env", ".env.staging", "packages/web/.env.development"], - "external": [ - { "type": "properties", "path": "~/.gradle/gradle.properties", - "keys": ["beaconClerkPkTest"] }, - { "type": "file", "path": "~/keystores/upload.keystore" } - ] -} -``` - -You rarely write it by hand: - -- **Auto-add (default):** `secrets push` discovers conventional files (`.env`, `.env.*`, `.dev.vars` — plus `package.json` workspace dirs once a manifest exists) and adds them to the manifest with an `==>` notice. Commit the manifest so other machines pick it up. -- **Explicit mode:** set `"options": {"autoAdd": false}` (a committed, team-shared setting) and `push` only syncs declared entries, warning about undeclared files. `secrets add ` is then the only manifest writer. Per-invocation: `push --frozen` (declared-only once) and `push --dry-run` (preview). -- `dotenv` paths are project-relative — nested monorepo paths like `packages/@acme/web/.env` are welcome; `..`, absolute paths, and symlinked manifests are refused. -- On the other machine, `secrets pull` restores exactly what the committed manifest declares, creating nested directories as needed. - -Projects without a manifest keep working exactly as before (and work without `jq`); the first `push` bootstraps one for you. - ### secrets run `secrets run` is a **pull → run → clear** pipeline: it runs `secrets pull` to decrypt the latest files into your project, executes your command, then runs `secrets clear` when that command finishes. Plaintext `.env` / `.dev.vars` files exist only while your command is running. @@ -425,10 +326,10 @@ git commit -am "switch to personal secrets" ### Monorepo support -For projects with multiple packages, add the `-w` flag to operate on all workspaces at once: +For projects with multiple packages (monorepos using `package.json` workspaces), add the `-w` flag to operate on all workspaces at once: ```bash -cd ~/myapp # npm/yarn workspaces, or a pnpm-workspace.yaml +cd ~/myapp # has package.json with "workspaces": ["apps/*", "packages/*"] secrets push -w # encrypts secrets from root + each workspace secrets pull -w # decrypts into root + each workspace directory secrets clear -w # clears secrets from root + each workspace @@ -445,61 +346,31 @@ Inside `~/.secrets/`, workspace secrets are organized by path: apps/api/.env.age # api workspace ``` -**Where workspaces are declared.** All three package managers are supported: - -| Manager | Declaration | -|---|---| -| npm | `package.json` → `"workspaces": ["apps/*"]` | -| yarn | `package.json` → `"workspaces": {"packages": ["apps/*"]}` | -| pnpm | `pnpm-workspace.yaml` → `packages:` block | - -`package.json` wins when it declares any workspaces; `pnpm-workspace.yaml` is -the fallback. Reading `package.json` requires `jq` (`brew install jq`); a -pnpm-only repo needs no jq for workspace discovery. - -If a root *looks* like a monorepo (a `pnpm-workspace.yaml` or a `packages/` -directory) but no workspace packages can be read from it, `secrets push` says -so on stderr rather than silently discovering nothing — that silence -previously made a pnpm repo indistinguishable from one with nothing to sync. +Requires `jq` (`brew install jq`). ### External files (Gradle properties) Some credentials don't live in your project at all. Android builds, for example, read keys from `~/.gradle/gradle.properties` — a global file, outside any project, shared by every Gradle project on your machine (the project's own `gradle.properties` is git-tracked, so it's the wrong home for secrets). `secrets` can sync specific keys from such a file without touching the unrelated keys around them. -You declare what to sync in the `external` array of your committed `.secrets.json`: +You declare what to sync in a committed `.secrets-files` manifest at your project root, one entry per line: -```json -{ - "version": 2, - "external": [ - { "type": "properties", "path": "~/.gradle/gradle.properties", - "keys": ["beaconClerkPkTest", "beaconClerkPkLive"] } - ] -} +``` +# +gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest beaconClerkPkLive ``` -- **type** — `properties` (sync named keys from a Java-properties-style file) or `file` (sync the whole file — see below). -- **path** — absolute or `~/`-relative; must resolve inside `$HOME`. For `properties` the basename must end in `.properties`. -- **keys** — the property names to sync (`properties` only). Only these keys are read on push and merged on pull; everything else in the file is left alone. `file` entries take no keys. - -> **Legacy `.secrets-files`:** older projects declared these entries in a line-based `.secrets-files`. It still parses, and the next `secrets push` absorbs its entries into `.secrets.json` (type `gradle-properties` becomes `properties`) — after that the legacy file is superseded and can be deleted. +- **type** — `gradle-properties` (the only supported type today). +- **path** — absolute or `~/`-relative. The basename must be `gradle.properties` and must resolve inside `$HOME`. +- **keys** — the property names to sync. Only these keys are read on push and merged on pull; everything else in the file is left alone. #### Syncing to a second machine -On the machine that already has the keys set, add the entry to `.secrets.json` (create the file if the project doesn't have one yet): +On the machine that already has the keys set: ```bash cd ~/myapp -cat > .secrets.json <<'EOF' -{ - "version": 2, - "external": [ - { "type": "properties", "path": "~/.gradle/gradle.properties", - "keys": ["beaconClerkPkTest", "beaconClerkPkLive"] } - ] -} -EOF -git add .secrets.json && git commit -m "sync gradle Clerk keys" +echo "gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest beaconClerkPkLive" > .secrets-files +git add .secrets-files && git commit -m "sync gradle Clerk keys" secrets push # ==> Extracted 2 key(s) from ~/.gradle/gradle.properties ``` @@ -518,74 +389,6 @@ secrets pull > **Note:** unlike `.env` files, merged Gradle keys are written as **permanent plaintext** into the target file — `secrets clear` does **not** remove them. This is appropriate for publishable / low-secrecy values (like Clerk publishable keys, `pk_*`). For high-value secrets that should never sit on disk, use `secrets run` with a `.env` instead. -#### Whole files (`file` type) - -Some external secrets are whole binary files — an Android upload keystore, a certificate. The `file` type syncs the file verbatim (binary-safe, encrypted with age like everything else): - -```json -{ "type": "file", "path": "~/keystores/beacon-upload.keystore" } -``` - -On `secrets push` the file is encrypted into `/external/`. On `secrets pull` it is restored to the same path with mode `600`; if a different version already exists there, it is backed up to `.secrets-bak` first. The same path rules apply (inside `$HOME`, no `..`, no symlinks). Like merged Gradle keys, restored files are permanent plaintext on disk — `secrets clear` does not remove them. - -### Onboarding and offboarding teammates - -By default every team member uses the **same** `key.txt` (one shared private key). The multi-recipient feature lets each teammate have their **own** keypair while still sharing one store — so you never hand out a secret key to a new hire, and removing an ex-teammate's access is one command. - -#### Onboarding a teammate - -```bash -# 1. Teammate generates their own keypair on their machine (never shares the private key) -age-keygen -o ~/.secrets/key.txt # writes key.txt; prints the public key - -# 2. Teammate sends you their PUBLIC key (printed by age-keygen, starts with age1…) -# — over Slack, email, whatever. Public keys are not secret. - -# 3. An existing member adds the public key to the store -secrets recipients add age1theirpublickey --name alice -# => Adds alice to recipients.txt, re-encrypts every blob to the full set, pushes. - -# 4. Teammate clones the store repo and drops their key.txt in place -git clone git@github.com:/my-secrets.git ~/.secrets -# (key.txt already generated in step 1 — nothing to copy) - -# 5. Teammate pulls into any project -cd ~/myapp -secrets pull -# => Their key matches one recipient stanza in every blob — it just works. -``` - -Run `secrets recipients list` to confirm who has access: - -``` -alice age1theirpublickey… -you age1yourpublickey… -``` - -#### Offboarding a teammate - -```bash -# Remove the recipient by name (or public key) and re-encrypt the store -secrets recipients rm alice -# => Removes alice from recipients.txt, re-encrypts every blob, pushes. -# Existing blobs are re-encrypted; the removed key can no longer decrypt them. -``` - -> **Important:** git history can't be un-shared. If alice had access during a period when genuinely sensitive values were stored, rotate those values now (update them in the external system and run `secrets push`). The re-encrypt prevents future access; history is permanent. - -#### Managing recipients - -```bash -secrets recipients list # show all recipient keys and names -secrets recipients add age1… # add a key (bootstraps recipients.txt on a legacy store) -secrets recipients add age1… --name bob # attach a human-readable label -secrets recipients rm bob # remove by name -secrets recipients rm age1… # remove by public key -secrets reencrypt # re-encrypt to current recipients (idempotent heal) -``` - -`secrets which` shows a `recipients: N (alice, bob, …)` line so you can always confirm the active set from any project directory. - ## Safety features - **`secrets run` auto-clears** — plaintext files are deleted when the command exits, errors, or is interrupted with Ctrl-C @@ -601,15 +404,13 @@ If you suspect your key has been compromised, or a teammate leaves the team: secrets rekey ``` -This generates a new key and re-encrypts all secrets (including `.env` and other dotfiles). After rekeying: +This generates a new key and re-encrypts all secrets. After rekeying: 1. Copy the new `~/.secrets/key.txt` to every machine and teammate 2. Old encrypted files remain in git history (encrypted with the old key, which should be discarded) For complete rotation with no historical exposure, create a fresh `~/.secrets/` repo. -> **Recovering from a broken rekey (pre-0.2.1.0):** Older versions of `rekey` skipped dotfiles (`.env`, `.dev.vars`) when re-encrypting, leaving their blobs on the *old* key while `key.txt` was replaced. If `secrets pull` now fails with `no identity matched any of the recipients`, those blobs are still encrypted to a key you no longer have. Restore the **old** `key.txt` from another machine that hasn't rekeyed, `secrets pull` to recover the plaintext, then `secrets rekey` again on 0.2.1.0 or later. - ## Environment variables | Variable | Default | Purpose | @@ -622,31 +423,16 @@ For complete rotation with no historical exposure, create a fresh `~/.secrets/` **"Not initialized"** — Run `secrets init` to create the `~/.secrets/` directory. -**"Found an existing key ... but no repo"** — You copied `key.txt` into `~/.secrets` and then ran `secrets init`. On a second machine you should *clone* your existing secrets repo, not re-initialize it (`init` is only for the very first machine). The error prints the exact `git clone` command to run — copy-paste it, or see [Additional machines](#additional-machines). When your project's `.secrets-store` file declares a remote URL, the command is filled in with the real URL. - **"No secret files found"** — You're in a directory that doesn't have `.env`, `.env.*`, or `.dev.vars` files. Make sure you're in the right project directory. **"Project not found"** — The project name doesn't match anything in `~/.secrets/`. Run `secrets list` to see what's stored. The name is usually derived from your directory name or git remote. **"Fast-forward pull failed"** — Someone else pushed secrets while you had local changes. Run `secrets pull` first, then retry your push. -**".secrets.json: invalid JSON" / "manifest version N is not supported"** — The committed manifest is malformed or written by a newer `secrets`. The error names the file; fix the syntax, or update the tool (`git pull` in the tool's clone). - -**"'jq' is not installed"** — Manifest features need `jq`. The error prints the install command for your platform. Manifest-less projects work without it. - -**Not sure the store is intact?** — Run `secrets verify` in a project to check its `.secrets.json` against the store (declared-but-missing blobs and orphaned blobs) and decrypt-test every blob with your current key. Use `secrets verify --all` for a store-wide decrypt sweep across every project. It's read-only — plaintext is streamed to `/dev/null`, never written to disk — and exits non-zero if anything is wrong, so it's safe to run in CI. - ## Development ```bash -# Run the test suite (272 tests across four files) +# Run the test suite (113 tests) brew install bats-core -bats test/ - -# Security regression subset — operator-local only (attack-payload fixtures). -# Required before ship; records sign-off in .gstack/security-signoff.json. -./test/run-security.sh +bats test/secrets.bats ``` - -Hosted AI agents must not run the security script or perform red-team/adversarial -review on this repo — see `.ship-policy.json` and `CLAUDE.md`. diff --git a/VERSION b/VERSION index d5ba5b5..e396b40 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.7.7.0 +0.2.0.0 diff --git a/docs/superpowers/plans/2026-06-08-additive-v2-dual-write.md b/docs/superpowers/plans/2026-06-08-additive-v2-dual-write.md deleted file mode 100644 index 4b21c0c..0000000 --- a/docs/superpowers/plans/2026-06-08-additive-v2-dual-write.md +++ /dev/null @@ -1,587 +0,0 @@ -# Additive-v2 Dual-Write Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Make store-format v2 *additive* — upgraded `secrets` clients read either blob suffix and dual-write existing `properties` externals — so the destructive `migrate --finalize` becomes optional GC and the cross-machine coordination gate disappears. - -**Architecture:** Replace the marker-driven single-suffix helper `_external_blob_suffix(type)` with two helpers: a read-resolver that tries `.properties.age` then falls back to `.gradle-properties.age`, and a write-targets helper that writes the v2 suffix always plus the v1 suffix *only when a v1 twin already exists* (dual-write existing externals; brand-new externals are v2-only — the intended forcing function). Reads and writes no longer depend on the `.secrets-format` marker, which keeps its meaning (born-v2 / finalized). Because fresh pushes now write the v2 suffix on any store, the migrate/finalize test fixtures (which relied on push producing a v1 blob) are updated to fabricate an old-client v1 blob. - -**Tech Stack:** Single bash 3.2 script (`secrets`); `age`, `git`, `jq`. Tests: `bats-core` (`test/migrate.bats`, `test/manifest.bats`). Every standalone `[[ ]]` test assertion ends with `|| false` (bash 3.2 ERR-trap gotcha). - -**Spec:** `docs/superpowers/specs/2026-06-08-additive-v2-dual-write-design.md`. **Ticket:** EGB-712. - ---- - -## Background facts (verified against branch `brian/egb-712-...`, post-EGB-710) - -- `_external_blob_suffix()` — `secrets:548-555`. Returns `properties` for `gradle-properties` on a v2 store (`_store_format == 2`), else the type verbatim; `file` always returns `file`. Blob path = `external/..age`. -- Callers of `_external_blob_suffix`: push file write `secrets:655`, push properties write `secrets:684`, pull read `secrets:713`, verify read `secrets:2056`. All four are replaced; then the helper is deleted. -- `_secrets_files_slug(path)` (`secrets:512`) derives the machine-independent slug. dotenv and `file` blobs are byte-identical in v1/v2 (only the `properties` suffix diverges). -- `_store_format()` (`secrets:529`) and the marker stay as-is — set only by `init` (born-v2) and `migrate --finalize`. Push must NOT stamp it (see spec §3). -- Existing test that codifies OLD write behavior: `test/migrate.bats:52` "push on a v1 store still writes .gradle-properties.age (back-compat)" — rewritten in Task 2. -- Tests that fabricate-or-rely-on a v1 properties blob from `make_v1_store; push` and break once push writes v2-only (repaired in Task 3): the migrate copy-forward/idempotent tests, the `--dry-run` rename test, the dotenv+file untouched test, all four finalize tests, the EGB-710 manifest-free / undeclared-twin tests, and the two `--status` tests that need an actual v1 blob. (dotenv-only and already-v2 tests are unaffected.) -- Baseline before this plan: `bats test/` = 242 passing. - -## File structure - -- Modify: `secrets` — delete `_external_blob_suffix` (548-555); add `_resolve_external_blob_read` + `_external_blob_write_targets` in its place; rewire push (651-688), pull (712-713), verify (2052-2068); extend `_migrate_status`; docs in `cmd_help`. -- Modify: `test/migrate.bats` — add `m_fake_v1_blob` helper; add read-fallback + write-rule + status-coverage tests; repair the v1-blob-dependent fixtures. -- Modify: `CLAUDE.md`, `README.md`, `VERSION` (→ `0.7.0.0`), `CHANGELOG.md`. - ---- - -## Task 1: Read-resolver — reads try both suffixes - -**Files:** `secrets` (replace `_external_blob_suffix` with the read-resolver; rewire pull + verify reads), `test/migrate.bats`. - -- [ ] **Step 1: Write the failing test** — a v2 store whose properties blob exists ONLY in the v1 suffix must still pull. - -Add to `test/migrate.bats` (after the format-marker tests, ~line 60): - -```bash -@test "pull reads a v1-suffix properties blob on a v2 store (read-fallback)" { - init_with_remote # born-v2 store (marker=2) - m_gradle_src $'beaconClerkPkTest=pk_test_v1\n' - create_project_dir rffallback - printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files - "$SECRETS_BIN" push rffallback >/dev/null 2>&1 # writes .properties.age on a v2 store - # Simulate an external that exists only in the v1 suffix (an old client wrote it): - local v2blob; v2blob=$(ls "$SECRETS_DIR/rffallback/external/"*.properties.age) - mv "$v2blob" "${v2blob%.properties.age}.gradle-properties.age" - rm -f "$HOME/.gradle/gradle.properties" - "$SECRETS_BIN" pull rffallback >/dev/null 2>&1 - run grep -q 'beaconClerkPkTest=pk_test_v1' "$HOME/.gradle/gradle.properties" - [ "$status" -eq 0 ] -} -``` - -- [ ] **Step 2: Run it, confirm it fails** - -Run: `bats test/migrate.bats -f "read-fallback"` -Expected: FAIL — old pull (`secrets:713`) uses `_external_blob_suffix gradle-properties` = `properties` on a v2 store, looks only for `.properties.age` (which we renamed away), warns "no encrypted data", restores nothing → the grep fails. - -- [ ] **Step 3: Replace `_external_blob_suffix` with the read-resolver.** Replace `secrets:542-555` (the comment block + `_external_blob_suffix()` through its closing `}`) with: - -```bash -# Resolve the on-disk path of an external blob for READING. Tries the v2 suffix -# (.properties.age) first, then falls back to the v1 (.gradle-properties.age) for -# `properties` externals, so an upgraded client finds the blob whichever format -# wrote it (additive v2 — EGB-712). `file` externals share one suffix in both -# formats. Echoes the path of the blob that exists; if neither exists, echoes the -# canonical v2 path so the caller's "no blob" message reads sensibly. Read-only. -_resolve_external_blob_read() { - local project="$1" slug="$2" mtype="$3" - local base="$SECRETS_DIR/$project/external/$slug" - case "$mtype" in - file) - echo "$base.file.age" ;; - properties|gradle-properties) - if [ -f "$base.properties.age" ]; then - echo "$base.properties.age" - elif [ -f "$base.gradle-properties.age" ]; then - echo "$base.gradle-properties.age" - else - echo "$base.properties.age" - fi ;; - *) - echo "$base.$mtype.age" ;; - esac -} -``` - -(The write-targets helper is added in Task 2 — leave a gap; do not reintroduce `_external_blob_suffix`.) - -- [ ] **Step 4: Rewire the pull read.** At `secrets:712-713`, replace: - -```bash - local slug; slug=$(_secrets_files_slug "$mpath") - local blob="$SECRETS_DIR/$project/external/$slug.$(_external_blob_suffix "$mtype").age" -``` - -with: - -```bash - local slug; slug=$(_secrets_files_slug "$mpath") - local blob; blob=$(_resolve_external_blob_read "$project" "$slug" "$mtype") -``` - -- [ ] **Step 5: Rewire the verify read.** At `secrets:2055-2065`, replace: - -```bash - slug=$(_secrets_files_slug "$epath") - erel="external/$slug.$(_external_blob_suffix "$etype").age" - # Account for BOTH the v1 and v2 suffix forms in the orphan set. During the - # migration window (after copy-forward, before --finalize) the v2 twin - # coexists with the v1 blob; neither should read as an orphan whichever - # format the store currently reports. (file's two forms are identical.) - expected="${expected}external/$slug.$etype.age"$'\n' - [ "$etype" = "gradle-properties" ] && expected="${expected}external/$slug.properties.age"$'\n' - eblob="$pdir/$erel" - if [ ! -f "$eblob" ]; then - echo "FINDING: external '$epath' ($etype) is declared but has no blob in the store ($project/$erel missing). Run 'secrets push'." >&2 -``` - -with: - -```bash - slug=$(_secrets_files_slug "$epath") - # Account for BOTH suffix forms in the orphan set — a dual-written `properties` - # external (additive v2 — EGB-712) legitimately has both blobs on disk; neither - # is an orphan. (file's two forms are identical.) - expected="${expected}external/$slug.$etype.age"$'\n' - [ "$etype" = "gradle-properties" ] && expected="${expected}external/$slug.properties.age"$'\n' - eblob=$(_resolve_external_blob_read "$project" "$slug" "$etype") - erel="${eblob#"$pdir"/}" - if [ ! -f "$eblob" ]; then - echo "FINDING: external '$epath' ($etype) is declared but has no blob in the store ($project/$erel missing). Run 'secrets push'." >&2 -``` - -(Note: `_external_blob_suffix` still has two remaining callers in push — push isn't rewired until Task 2, so the script still parses and runs. Those calls keep working because the function is only deleted in Task 2 Step 6, after push is rewired.) - -**IMPORTANT:** do NOT delete `_external_blob_suffix` yet — push (`secrets:655`, `secrets:684`) still calls it until Task 2. Deleting it now breaks push. - -- [ ] **Step 6: Run the read-fallback test + full migrate suite** - -Run: `bats test/migrate.bats` -Expected: the new "read-fallback" test PASSES; all other migrate tests still PASS (push unchanged; pull/verify now use the resolver, which is equivalent to the old behavior whenever the suffix matches the store format). - -- [ ] **Step 7: Commit** - -```bash -git add secrets test/migrate.bats -git commit -m "feat: read-resolver tries both external suffixes (additive v2, EGB-712)" -``` - ---- - -## Task 2: Write-targets — twin rule (dual-write existing, v2-only for new) - -**Files:** `secrets` (add `_external_blob_write_targets`; rewire push file + properties writes; delete `_external_blob_suffix`), `test/migrate.bats`. - -- [ ] **Step 1: Write the failing tests.** Add to `test/migrate.bats`: - -```bash -@test "push writes the v2 suffix for a fresh external even on a v1 store" { - make_v1_store - m_gradle_src $'beaconClerkPkTest=pk_test_abc\n' - create_project_dir freshv1 - printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files - "$SECRETS_BIN" push freshv1 >/dev/null 2>&1 - run bash -c "ls $SECRETS_DIR/freshv1/external/*.properties.age" - [ "$status" -eq 0 ] # v2 suffix regardless of marker - run bash -c "ls $SECRETS_DIR/freshv1/external/*.gradle-properties.age 2>/dev/null" - [ "$status" -ne 0 ] # no v1 twin for a brand-new external -} - -@test "push dual-writes the v1 twin so old clients stay fresh" { - make_v1_store - m_gradle_src $'beaconClerkPkTest=pk_test_old\n' - create_project_dir dualwrite - printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files - "$SECRETS_BIN" push dualwrite >/dev/null 2>&1 # v2-only (fresh) - m_fake_v1_twin dualwrite # simulate a pre-existing v1 twin - m_gradle_src $'beaconClerkPkTest=pk_test_new\n' - "$SECRETS_BIN" push dualwrite >/dev/null 2>&1 # twin exists -> dual-write both - # Prove the v1 twin was refreshed: drop the v2 blob, pull, expect the NEW value. - rm -f "$SECRETS_DIR/dualwrite/external/"*.properties.age - rm -f "$HOME/.gradle/gradle.properties" - "$SECRETS_BIN" pull dualwrite >/dev/null 2>&1 - run grep -q 'beaconClerkPkTest=pk_test_new' "$HOME/.gradle/gradle.properties" - [ "$status" -eq 0 ] -} -``` - -- [ ] **Step 2: Add the `m_fake_v1_twin` test helper.** In `test/migrate.bats`, next to `make_v1_store` (~line 12), add: - -```bash -# Simulate an old (v1) client's properties blob: copy the pushed v2 -# .properties.age to its v1 .gradle-properties.age twin. (Current clients never -# write the v1 suffix for a fresh external, so tests fabricate it.) Use `cp` to -# KEEP the v2 blob (dual present); see m_make_v1_only to leave only the v1 blob. -m_fake_v1_twin() { - local proj="$1" v2 - v2=$(ls "$SECRETS_DIR/$proj/external/"*.properties.age) - cp "$v2" "${v2%.properties.age}.gradle-properties.age" -} -# Like m_fake_v1_twin but renames (leaves ONLY the v1 blob) — for old-client-only -# / copy-forward fixtures. -m_make_v1_only() { - local proj="$1" v2 - v2=$(ls "$SECRETS_DIR/$proj/external/"*.properties.age) - mv "$v2" "${v2%.properties.age}.gradle-properties.age" -} -``` - -- [ ] **Step 3: Run the new tests, confirm they fail** - -Run: `bats test/migrate.bats -f "fresh external even on a v1 store"` -Expected: FAIL — on a v1 store, the old push (`_external_blob_suffix gradle-properties` = `gradle-properties`) writes `.gradle-properties.age`, so the `*.properties.age` assertion fails. -Run: `bats test/migrate.bats -f "dual-writes the v1 twin"` -Expected: FAIL — old push writes a single suffix; the second push won't refresh the v1 twin. - -- [ ] **Step 4: Add the write-targets helper.** Immediately after `_resolve_external_blob_read` (added in Task 1), insert: - -```bash -# The on-disk path(s) to WRITE for an external blob, one per line. For a -# `properties` external this is the v2 suffix (.properties.age) ALWAYS, plus the -# v1 suffix (.gradle-properties.age) WHEN a v1 twin already exists in the store -# (dual-write keeps old clients fresh; a brand-new external is v2-only — the -# intended forcing function, additive v2 / EGB-712). `file` externals have a -# single suffix in both formats. Independent of the store marker. -_external_blob_write_targets() { - local project="$1" slug="$2" mtype="$3" - local base="$SECRETS_DIR/$project/external/$slug" - case "$mtype" in - file) - echo "$base.file.age" ;; - properties|gradle-properties) - echo "$base.properties.age" - [ -f "$base.gradle-properties.age" ] && echo "$base.gradle-properties.age" ;; - *) - echo "$base.$mtype.age" ;; - esac -} -``` - -- [ ] **Step 5: Rewire the push writes.** At `secrets:651-658` (the `file` branch), replace: - -```bash - if [ "$mtype" = "file" ]; then - # EGB-652: whole-file sync — encrypt the file verbatim (binary-safe). - mkdir -p "$SECRETS_DIR/$project/external" - local fslug; fslug=$(_secrets_files_slug "$mpath") - age -r "$pubkey" -o "$SECRETS_DIR/$project/external/$fslug.$(_external_blob_suffix file).age" "$expanded" - info "Encrypted file $mpath" - pushed=$((pushed + 1)) - continue - fi -``` - -with: - -```bash - if [ "$mtype" = "file" ]; then - # EGB-652: whole-file sync — encrypt the file verbatim (binary-safe). - mkdir -p "$SECRETS_DIR/$project/external" - local fslug; fslug=$(_secrets_files_slug "$mpath") - local wt - while IFS= read -r wt; do - [ -n "$wt" ] || continue - age -r "$pubkey" -o "$wt" "$expanded" - done < <(_external_blob_write_targets "$project" "$fslug" file) - info "Encrypted file $mpath" - pushed=$((pushed + 1)) - continue - fi -``` - -Then at `secrets:682-684` (the properties branch), replace: - -```bash - mkdir -p "$SECRETS_DIR/$project/external" - local slug; slug=$(_secrets_files_slug "$mpath") - age -r "$pubkey" -o "$SECRETS_DIR/$project/external/$slug.$(_external_blob_suffix "$mtype").age" "$tmp" -``` - -with: - -```bash - mkdir -p "$SECRETS_DIR/$project/external" - local slug; slug=$(_secrets_files_slug "$mpath") - local wt - while IFS= read -r wt; do - [ -n "$wt" ] || continue - age -r "$pubkey" -o "$wt" "$tmp" - done < <(_external_blob_write_targets "$project" "$slug" "$mtype") -``` - -- [ ] **Step 6: Delete the now-unused `_external_blob_suffix`.** Confirm zero remaining callers first: - -Run: `grep -n "_external_blob_suffix" secrets` -Expected: no matches (all four call sites rewired). If any remain, rewire them before deleting. Then delete the `_resolve_external_blob_read`-replaced... — it's already gone (replaced in Task 1). Verify the function is absent: `grep -c "_external_blob_suffix()" secrets` → `0`. - -- [ ] **Step 7: Run the two new write tests** - -Run: `bats test/migrate.bats -f "fresh external even on a v1 store"` then `-f "dual-writes the v1 twin"` -Expected: both PASS. - -- [ ] **Step 8: Rewrite the obsolete back-compat test.** Replace the `@test "push on a v1 store still writes .gradle-properties.age (back-compat)"` block (`test/migrate.bats:52-60`) with: - -```bash -@test "push on a v1 store writes the v2 suffix for a fresh external (additive v2)" { - make_v1_store - m_gradle_src $'beaconClerkPkTest=pk_test_abc\n' - create_project_dir v1push - printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files - "$SECRETS_BIN" push v1push >/dev/null 2>&1 - run bash -c "ls $SECRETS_DIR/v1push/external/*.properties.age" - [ "$status" -eq 0 ] -} -``` - -- [ ] **Step 9: Run the full migrate suite — expect the v1-blob-dependent fixtures to FAIL.** This is expected; Task 3 repairs them. - -Run: `bats test/migrate.bats` -Expected: the read-fallback + two write tests + rewritten back-compat test PASS; several copy-forward/finalize/status tests now FAIL (push no longer writes a `.gradle-properties.age` for them to migrate). Note which fail — Task 3 fixes exactly those. - -- [ ] **Step 10: Commit** (suite intentionally not yet fully green — Task 3 follows immediately) - -```bash -git add secrets test/migrate.bats -git commit -m "feat: twin-rule write targets — dual-write existing, v2-only for new (additive v2, EGB-712)" -``` - ---- - -## Task 3: Repair migrate/finalize/status fixtures - -**Files:** `test/migrate.bats`. No production code changes — this re-greens the suite by fabricating the old-client v1 blobs that push no longer writes. - -The rule for each repair: after the `"$SECRETS_BIN" push ` line, insert a fabrication call: -- Use **`m_make_v1_only `** (rename → only the v1 blob exists) for tests asserting a `*.gradle-properties.age` blob exists / is copy-forwarded (mirrors the pre-EGB-712 state where push produced a v1 blob). -- Use **`m_fake_v1_twin `** (keep both) only where a test needs both suffixes present. - -- [ ] **Step 1: Repair the copy-forward / dry-run / idempotent tests.** In each of these tests, insert `m_make_v1_only ` immediately after the `push ` line: - - `"migrate --dry-run reports the rename and writes nothing"` (proj `dryproj`) - - `"migrate copy-forward creates the v2 twin and keeps the v1 blob (byte-identical)"` (proj `cfproj`) - - `"migrate copy-forward is idempotent"` (proj `idemproj`) - - `"migrate leaves dotenv and file blobs untouched"` (proj `mixproj`) — note this one ALSO pushes a `file` external; `m_make_v1_only` only touches `*.properties.age`, leaving the `.file.age` blob alone (correct). - -Worked example — the copy-forward test becomes: - -```bash -@test "migrate copy-forward creates the v2 twin and keeps the v1 blob (byte-identical)" { - make_v1_store - m_gradle_src $'beaconClerkPkTest=pk_test_abc\n' - create_project_dir cfproj - printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files - "$SECRETS_BIN" push cfproj >/dev/null 2>&1 - m_make_v1_only cfproj - local old; old=$(ls "$SECRETS_DIR/cfproj/external/"*.gradle-properties.age) - run "$SECRETS_BIN" migrate - [ "$status" -eq 0 ] - local new; new=$(ls "$SECRETS_DIR/cfproj/external/"*.properties.age) - [ -f "$old" ] - [ -f "$new" ] - cmp -s "$old" "$new" -} -``` - -- [ ] **Step 2: Repair the finalize tests.** Insert `m_make_v1_only ` after the `push ` line in: - - `"finalize refuses when verify --all is not green"` (proj `failverify`) — then the existing `migrate` step creates the twin; corrupting `*.properties.age` still trips verify. - - `"finalize refuses an un-twinned v1 blob (project not migrated)"` (proj `untwinned`) — leaves a lone v1 blob, no twin: exactly the un-twinned state the test wants. - - `"finalize green path drops v1, keeps v2, stamps the marker"` (proj `finproj`). - - `"finalize cuts a recovery tag before deleting v1 blobs"` (proj — read it from the test). - -- [ ] **Step 3: Repair the EGB-710 manifest-free / undeclared-twin tests.** Insert `m_make_v1_only ` after the `push ` line in: - - `"migrate copy-forwards a v1 properties blob with no .secrets.json (manifest-free)"` (proj `nomanifestblob`) — insert BEFORE the `rm -f .secrets.json` line. - - `"migrate twins a store blob even when the manifest no longer declares it"` (proj `staleblob`) — insert before the `.secrets.json` rewrite. - -- [ ] **Step 4: Repair the `--status` tests that need a real v1 blob.** - - `"migrate --status flags a project that needs migrating"` (proj `needsmig`) — insert `m_make_v1_only needsmig` after push, so a lone un-twinned v1 blob exists → NEEDS MIGRATE. - - `"migrate --status reports finalize-ready once every blob is twinned"` (proj `readymig`) — insert `m_make_v1_only readymig` after push and BEFORE the `migrate` step (migrate then creates the twin → finalize-ready). - -- [ ] **Step 5: Run the full migrate suite** - -Run: `bats test/migrate.bats` -Expected: ALL pass. If any copy-forward test still reports "nothing to migrate", its `m_make_v1_only` call is missing or misplaced (must come after push, before migrate). - -- [ ] **Step 6: Run the WHOLE suite** (manifest.bats exercises externals end-to-end and must still be green) - -Run: `bats test/` -Expected: all pass. If a `manifest.bats` external test fails, check it isn't asserting a specific suffix that additive-v2 changed (a fresh push now writes `.properties.age`); update such an assertion the same way (assert `.properties.age`, or use the resolver-agnostic round-trip via pull). - -- [ ] **Step 7: Commit** - -```bash -git add test/migrate.bats -git commit -m "test: fabricate old-client v1 blobs in migrate/finalize/status fixtures (additive v2, EGB-712)" -``` - ---- - -## Task 4: `migrate --status` — dual-write coverage line - -**Files:** `secrets` (`_migrate_status`, `secrets:2190-2227`), `test/migrate.bats`. - -Adds visibility into which `properties` externals are v2-only (old clients can't read them — the forcing function) vs dual-written (old clients still served). - -- [ ] **Step 1: Write the failing test.** Add to `test/migrate.bats`: - -```bash -@test "migrate --status counts v2-only externals (old clients not served)" { - make_v1_store - m_gradle_src $'beaconClerkPkTest=pk_test_abc\n' - create_project_dir v2onlyext - printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files - "$SECRETS_BIN" push v2onlyext >/dev/null 2>&1 # v2-only (fresh, no v1 twin) - run "$SECRETS_BIN" migrate --status - [ "$status" -eq 0 ] # no v1 blobs -> finalize-ready - [[ "$output" == *"v2-only"* ]] || false # surfaced as v2-only coverage -} -``` - -- [ ] **Step 2: Run it, confirm it fails** - -Run: `bats test/migrate.bats -f "counts v2-only externals"` -Expected: FAIL — `_migrate_status` currently only counts `*.gradle-properties.age`; it never mentions `v2-only`. - -- [ ] **Step 3: Extend `_migrate_status`.** In `_migrate_status` (`secrets:2190`), inside the `for dir` loop, after the existing `while ... done < <(find "$dir" -type f -name '*.gradle-properties.age' ...)` block and before the per-project classification, add a v2-only count, then surface it in the per-project line. Concretely, replace the classification block: - -```bash - if [ "$v1" -eq 0 ]; then - echo " $project: v2-ready (no v1 properties blobs)" - elif [ "$untwinned" -eq 0 ]; then - echo " $project: migrated ($v1 v1 blob(s), all twinned)" - else - echo " $project: NEEDS MIGRATE ($untwinned of $v1 v1 blob(s) un-twinned) — cd into the project and run 'secrets migrate'" - any_untwinned=1 - fi -``` - -with: - -```bash - # v2-only externals: a .properties.age with no .gradle-properties.age twin — - # old (v1) clients cannot read these (the additive-v2 forcing function). - local v2only=0 pf - while IFS= read -r pf; do - [ -f "$pf" ] || continue - [ -f "${pf%.properties.age}.gradle-properties.age" ] || v2only=$((v2only + 1)) - done < <(find "$dir" -type f -name '*.properties.age' 2>/dev/null) - local v2note="" - [ "$v2only" -gt 0 ] && v2note=" [$v2only v2-only — old clients not served]" - if [ "$v1" -eq 0 ]; then - echo " $project: v2-ready (no v1 properties blobs)$v2note" - elif [ "$untwinned" -eq 0 ]; then - echo " $project: migrated ($v1 v1 blob(s), all twinned)$v2note" - else - echo " $project: NEEDS MIGRATE ($untwinned of $v1 v1 blob(s) un-twinned) — cd into the project and run 'secrets migrate'$v2note" - any_untwinned=1 - fi -``` - -(`local` inside the loop is bash-3.2-fine — it re-declares per iteration.) - -- [ ] **Step 4: Run the new test + status suite** - -Run: `bats test/migrate.bats -f "status"` -Expected: all status tests PASS, including the new v2-only one. - -- [ ] **Step 5: Run the full suite** - -Run: `bats test/` -Expected: all pass. - -- [ ] **Step 6: Commit** - -```bash -git add secrets test/migrate.bats -git commit -m "feat: migrate --status surfaces v2-only externals (coverage, EGB-712)" -``` - ---- - -## Task 5: Docs, help, version, changelog - -**Files:** `secrets` (`cmd_help`), `CLAUDE.md`, `README.md`, `VERSION`, `CHANGELOG.md`. - -- [ ] **Step 1: `cmd_help` — reframe finalize as optional.** In `cmd_help()`, replace the finalize line: - -``` - secrets migrate --finalize Drop v1 blobs and mark the store v2 (after verify) -``` - -with: - -``` - secrets migrate --finalize Optional GC: drop v1 blobs and mark the store pure v2 -``` - -- [ ] **Step 2: `CLAUDE.md` — additive-v2 paragraph.** In the "Store format" bullet (line ~65), after the migration-chain sentence, add (new sentence, same bullet): - -``` - Additive v2 (EGB-712): upgraded clients read either blob suffix (`_resolve_external_blob_read` tries `.properties.age` then `.gradle-properties.age`) and dual-write a `properties` external only when a v1 twin already exists (`_external_blob_write_targets`) — so existing externals keep old clients fresh, brand-new externals are v2-only (a gentle forcing function), and `migrate --finalize` is now OPTIONAL GC rather than a required, coordination-gated flag-day. dotenv and `file` blobs are identical across formats, so they always propagate to old clients. -``` - -- [ ] **Step 3: `README.md` — propagation table.** Add, near the migrate rows in the command table or in a short "Upgrading / store format" subsection, the propagation-by-secret-type summary (verbatim from spec §6): - -```markdown -**Do teammates on an older `secrets` get new secrets?** - -| Secret type | Old client gets it? | -|---|---| -| `.env` / `.env.*` / `.dev.vars` | **Yes, always** (blob name identical across formats) | -| whole-file external | **Yes, always** | -| `properties` external that already existed | **Yes** (dual-written) | -| brand-new `properties` external | **No — must upgrade `secrets`** (the forcing function) | - -"Upgrade your secrets" = `git pull` the tool clone (binary ≥ 0.6.0.0) and/or `secrets migrate` the store. A read-only teammate only needs the tool `git pull`. -``` - -- [ ] **Step 4: `VERSION`** — set to `0.7.0.0`. - -- [ ] **Step 5: `CHANGELOG.md`** — insert above the top entry: - -```markdown -## [0.7.0.0] - 2026-06-08 - -### Changed - -- **Additive store-format v2 (EGB-712)** — upgraded `secrets` clients now read - either external blob suffix (`.properties.age` or the legacy - `.gradle-properties.age`) and **dual-write** a `properties` external whenever a - v1 twin already exists in the store. Existing externals keep working for - teammates on an older `secrets`; only a brand-new `properties` external is - written v2-only (a gentle "upgrade to see it" forcing function). dotenv and - whole-`file` externals are unchanged across formats and always propagate. -- **`secrets migrate --finalize` is now optional GC**, not a required milestone. - Because clients dual-write and read-fall-back, no teammate is ever cut off by - *not* finalizing; finalize only reclaims the duplicate v1 blobs, and stays - deferrable indefinitely. Its safety gates are unchanged. - -### Added - -- **`secrets migrate --status`** now reports `v2-only` externals per project - (the ones an un-upgraded client cannot read), so you can see the forcing - function's footprint at a glance. -``` - -- [ ] **Step 6: Run the full suite** - -Run: `bats test/` -Expected: all pass (docs don't affect tests). Confirm the count is baseline 242 + net new tests from Tasks 1/2/4 (read-fallback, two write tests, v2-only status) minus the rewritten back-compat test (replaced, not added) = **246**. - -- [ ] **Step 7: Sanity — help renders** - -Run: `./secrets help 2>&1 | grep -- "--finalize"` -Expected: shows the reframed "Optional GC" line. (Help only prints; touches no store.) - -- [ ] **Step 8: Commit** - -```bash -git add secrets CLAUDE.md README.md VERSION CHANGELOG.md -git commit -m "docs: additive-v2 propagation + optional-GC finalize; bump 0.7.0.0 (EGB-712)" -``` - ---- - -## Self-review against the spec - -- **§1 read resolution** → Task 1 (`_resolve_external_blob_read`, wired into pull + verify). Test: read-fallback. -- **§2 write rule / twin rule** → Task 2 (`_external_blob_write_targets`, push wiring). Tests: fresh-→v2-only, existing-twin-→dual-write. -- **§3 marker NOT stamped** → no production change (push never touches the marker); guarded implicitly by Task 3 keeping the `make_v1_store; push; migrate` flow working (store stays markerless after push). The plan deliberately does not add marker-stamping. -- **§4 finalize = optional GC** → unchanged logic; reframed in Task 5 docs/help. Existing finalize tests stay green (Task 3 keeps them green). -- **§5 migrate / --status coverage** → Task 4 (v2-only line); `migrate` copy-forward unchanged (EGB-710). -- **§6 propagation table** → Task 5 README/CLAUDE.md. -- **§7 caveat** → documented in CHANGELOG/CLAUDE.md framing; the read-fallback test exercises the v1-only read path. -- **§8 back-compat matrix** → covered across Task 1 (reads), Task 2 (writes), Task 3 (old-client v1 blobs simulated). -- **Safety invariants** → no path-rail changes; bash 3.2 (no associative arrays; `while read` + `find`); recursive walks unchanged; finalize gating untouched. - -**Type/name consistency:** `_resolve_external_blob_read(project, slug, mtype)` and `_external_blob_write_targets(project, slug, mtype)` use the same arg order everywhere; test helpers `m_fake_v1_twin` (keep both) and `m_make_v1_only` (rename to v1-only) are used consistently per their documented semantics. - -**Placeholder scan:** none — every step carries verbatim code or an exact enumerated edit with the precise insertion point. - -## Operator-local follow-up (not part of this plan) - -Per `.ship-policy.json`, before any PR ask the operator to run `./test/run-security.sh` and complete the SIGNOFF. EGB-712 also needs its one stale AC bullet ("Marker auto-stamps…") corrected to match the §3 decision (no auto-stamp) — a one-line Linear edit. diff --git a/docs/superpowers/plans/2026-06-08-egb-710-migrate-guided-flow.md b/docs/superpowers/plans/2026-06-08-egb-710-migrate-guided-flow.md deleted file mode 100644 index b552428..0000000 --- a/docs/superpowers/plans/2026-06-08-egb-710-migrate-guided-flow.md +++ /dev/null @@ -1,447 +0,0 @@ -# EGB-710: `secrets migrate` guided flow (manifest-free copy-forward + `--status` survey) Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Replace the dead-end "No .secrets.json — cd into a project that has a manifest" error in `secrets migrate` with a manifest-free copy-forward that just works, plus a `secrets migrate --status` survey that tells the operator exactly which projects still need migrating. - -**Architecture:** The per-project copy-forward step (`_migrate_project`) currently reads `$PWD/.secrets.json` to find `properties` externals, then looks for their v1 blobs. This makes it die on a legacy `.secrets-files`-only project (the real EGB-710 repro) and — worse — silently skips any store blob the manifest doesn't declare, leaving it un-twinned so `--finalize` later refuses it. The fix: enumerate the **store** (`$SECRETS_DIR//external/*.gradle-properties.age`) directly and copy each to its `.properties.age` twin via suffix-swap — the *exact* logic `_migrate_finalize` already uses. No manifest needed, dotenv/file-only projects become a clean no-op, and migrate twins precisely the blobs finalize will demand twins for. A new `--status` mode walks every project dir and reports per-project readiness + whether the store is finalize-ready. - -**Tech Stack:** Single bash 3.2 script (`secrets`), `age`, `git`, `jq` (unaffected here). Tests: `bats-core` (`test/migrate.bats`). Every standalone `[[ ]]` assertion ends with `|| false` (bash 3.2 ERR-trap gotcha). - ---- - -## Background facts (verified against HEAD 633d19e) - -- `_migrate_project()` lives at `secrets:2135`; it `die`s at `secrets:2142-2148` on missing `$PWD/.secrets.json`, then iterates `_json_external_entries "$manifest"` filtered to `gradle-properties` (`secrets:2154-2174`). -- `_migrate_finalize()` (`secrets:2196`) already enumerates blobs manifest-free: `find "$SECRETS_DIR" -type f -name '*.gradle-properties.age'` and derives the twin as `new="${f%.gradle-properties.age}.properties.age"` (`secrets:2212-2217`). The per-project step will mirror this, scoped to one project dir. -- `cmd_migrate()` flag parser: `secrets:2267-2285`. -- `derive_project_name ""` (`secrets:101`) needs no manifest — it uses the git remote basename or `basename "$PWD"`. -- `info()`/`die()`: `secrets:36-37`. `ensure_store_protections` is called post-write in the existing copy-forward. -- Output-contract strings existing tests depend on (must be preserved): `"would migrate"` (migrate.bats:80), `"0 blob(s) would be copy-forwarded"` (migrate.bats:94), `"1 already present"` (migrate.bats:121, idempotent re-run prints `$already already present`), `"already format v2"` (migrate.bats:139). -- The one test that codifies the OLD dead-end — `"migrate with no manifest in cwd dies with a directed message"` (migrate.bats:126) — is the behavior we are intentionally changing; it gets rewritten in Task 1. -- No test asserts the non-dry-run "nothing to migrate" wording (grep confirmed), so that message is free to change. We keep the substring `no v1 properties blobs` regardless for safety. -- VERSION is `0.6.0.1`; bump to `0.6.1.0` (new subcommand flag + behavior change). `MANIFEST_VERSION` stays `2` (no schema change). - -## File structure - -- Modify: `secrets` — rewrite `_migrate_project` (`secrets:2135-2191`), add `_migrate_status`, extend `cmd_migrate` flag parsing + dispatch (`secrets:2267-2285`), update `cmd_help` migrate lines (`secrets:2308-2309`). -- Modify: `test/migrate.bats` — rewrite the dead-end test; add manifest-free, finalize-consistency, and `--status` tests. -- Modify: `CLAUDE.md` — update the migrate paragraph (Architecture → Store format) to note manifest-free copy-forward + `--status`. -- Modify: `README.md` — migrate usage/help. -- Modify: `VERSION` → `0.6.1.0`; `CHANGELOG.md` — new entry. - ---- - -## Task 1: Manifest-free per-project copy-forward - -**Files:** -- Modify: `secrets` — `_migrate_project()` (`secrets:2135-2191`) -- Test: `test/migrate.bats` - -- [ ] **Step 1: Write the failing test — migrate works with no `.secrets.json` (the EGB-710 repro)** - -Add to `test/migrate.bats` (after the existing copy-forward tests, ~line 124): - -```bash -@test "migrate copy-forwards a v1 properties blob with no .secrets.json (manifest-free)" { - # The EGB-710 repro: a legacy project has a v1 properties blob in the store - # but no .secrets.json (it predates the manifest). migrate must NOT dead-end. - make_v1_store - m_gradle_src $'beaconClerkPkTest=pk_test_abc\n' - create_project_dir nomanifestblob - printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files - "$SECRETS_BIN" push nomanifestblob >/dev/null 2>&1 - rm -f .secrets.json # simulate a pre-manifest project - run "$SECRETS_BIN" migrate - [ "$status" -eq 0 ] - run bash -c "ls $SECRETS_DIR/nomanifestblob/external/*.properties.age" - [ "$status" -eq 0 ] # v2 twin written despite no manifest - run bash -c "ls $SECRETS_DIR/nomanifestblob/external/*.gradle-properties.age" - [ "$status" -eq 0 ] # v1 kept (non-destructive) -} -``` - -- [ ] **Step 2: Run it to confirm it fails** - -Run: `bats test/migrate.bats -f "manifest-free"` -Expected: FAIL — current code `die`s with "No .secrets.json in …" (status 1), so the first `[ "$status" -eq 0 ]` fails. - -- [ ] **Step 3: Rewrite `_migrate_project` to enumerate the store, not the manifest** - -Replace the body of `_migrate_project()` (`secrets:2135-2191`, from `_migrate_project() {` through its closing `}`) with: - -```bash -_migrate_project() { - local dry_run="$1" - if [ "$(_store_format)" = "2" ]; then - info "Store is already format v2 — nothing to migrate." - return 0 - fi - local project; project=$(derive_project_name "") - local pdir="$SECRETS_DIR/$project" - - # Source of truth for the copy-forward is the STORE, not a project manifest. - # Every v1 properties blob is a `*.gradle-properties.age` file whose v2 twin - # is the same name with the `.properties.age` suffix (the only on-disk change - # v2 makes). Enumerating the store — exactly as `_migrate_finalize` does — - # means migrate twins precisely the blobs finalize will demand twins for, with - # no manifest dependency. This is why a legacy `.secrets-files`-only project - # (no `.secrets.json` yet) migrates cleanly instead of dead-ending, and why a - # store blob the manifest no longer declares still gets a twin. - local moved=0 already=0 would=0 v1count=0 f new - while IFS= read -r f; do - [ -f "$f" ] || continue - v1count=$((v1count + 1)) - new="${f%.gradle-properties.age}.properties.age" - if [ -f "$new" ]; then - already=$((already + 1)) - continue - fi - if [ "$dry_run" = true ]; then - echo "would migrate: ${f#"$SECRETS_DIR"/} -> $(basename "$new")" - would=$((would + 1)) - else - cp "$f" "$new" - moved=$((moved + 1)) - fi - done < <(find "$pdir/external" -type f -name '*.gradle-properties.age' 2>/dev/null) - - if [ "$dry_run" = true ]; then - echo "migrate --dry-run: $would blob(s) would be copy-forwarded for '$project' (writes nothing); $already already present. v1 blobs are kept until 'secrets migrate --finalize'." - return 0 - fi - if [ "$moved" -eq 0 ] && [ "$already" -eq 0 ]; then - info "Nothing to migrate for '$project' — no v1 properties blobs in the store (already v2-shaped). If you expected one, run 'secrets push' first, then re-run 'secrets migrate'." - return 0 - fi - ensure_store_protections - git -C "$SECRETS_DIR" add -A - git -C "$SECRETS_DIR" commit -m "migrate: copy-forward v2 twins for $project" >/dev/null 2>&1 || true - # Push the twins so a --finalize on another machine sees them (finalize - # refuses any v1 blob without a twin). Mirrors push/rekey's push behavior. - git -C "$SECRETS_DIR" remote get-url origin >/dev/null 2>&1 && git -C "$SECRETS_DIR" push >/dev/null 2>&1 || true - info "Copy-forward for '$project': $moved new v2 twin(s), $already already present. v1 blobs kept (non-destructive). Run 'secrets migrate --finalize' once every project is migrated and every machine is upgraded." -} -``` - -Notes for the implementer: -- This removes the only in-script caller of `_check_manifest_file`/`_json_external_entries` *inside migrate*; both remain defined and used by push/pull/verify, so do **not** delete them. -- `${f#"$SECRETS_DIR"/}` keeps the dry-run line's store-relative form (matches the old `$project/external/...` style closely enough; the test only checks the substring `would migrate`). -- The `moved==0 && already==0` branch keeps the substring `no v1 properties blobs`. The idempotent re-run path (`moved==0, already>0`) falls through to the final `info` printing `$already already present`, preserving the `1 already present` contract. - -- [ ] **Step 4: Run the new test + the full migrate suite** - -Run: `bats test/migrate.bats` -Expected: the new "manifest-free" test PASSES; existing dry-run/copy-forward/idempotent/finalize tests still PASS; the "migrate with no manifest in cwd dies" test (migrate.bats:126) now FAILS (we fix it in Step 5). - -- [ ] **Step 5: Update the test that codified the old dead-end** - -Replace the test at `test/migrate.bats:126-132` (`"migrate with no manifest in cwd dies with a directed message"`) with: - -```bash -@test "migrate in a project with no manifest and no store blobs is a clean no-op" { - make_v1_store - local dir="$WORK_DIR/nomanifest"; mkdir -p "$dir"; cd "$dir" - run "$SECRETS_BIN" migrate - [ "$status" -eq 0 ] - [[ "$output" == *"no v1 properties blobs"* ]] || false -} -``` - -- [ ] **Step 6: Run the full suite to confirm green** - -Run: `bats test/migrate.bats` -Expected: all PASS. - -- [ ] **Step 7: Commit** - -```bash -git add secrets test/migrate.bats -git commit -m "fix: migrate copy-forward is manifest-free, no dead-end on legacy projects (EGB-710)" -``` - ---- - -## Task 2: Finalize-consistency regression test (store blob not in manifest) - -**Files:** -- Test: `test/migrate.bats` - -This proves the latent-bug fix: the old manifest-driven migrate skipped store blobs the manifest didn't declare, leaving them un-twinned so `--finalize` refused them. The rewrite twins them. - -- [ ] **Step 1: Write the test** - -Add to `test/migrate.bats`: - -```bash -@test "migrate twins a store blob even when the manifest no longer declares it" { - make_v1_store - m_gradle_src $'beaconClerkPkTest=pk_test_abc\n' - create_project_dir staleblob - printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files - "$SECRETS_BIN" push staleblob >/dev/null 2>&1 - # The blob is now in the store. Drop the external from the project's manifest - # entirely (and remove the legacy file) so NO manifest declares it. - printf '{"version":2,"dotenv":[".env",".env.staging"]}\n' > .secrets.json - rm -f .secrets-files - run "$SECRETS_BIN" migrate - [ "$status" -eq 0 ] - run bash -c "ls $SECRETS_DIR/staleblob/external/*.properties.age" - [ "$status" -eq 0 ] # twinned despite not being declared anywhere -} -``` - -- [ ] **Step 2: Run it** - -Run: `bats test/migrate.bats -f "no longer declares"` -Expected: PASS (the Task 1 rewrite already makes this green — this test guards against regressing back to manifest-driven enumeration). - -- [ ] **Step 3: Commit** - -```bash -git add test/migrate.bats -git commit -m "test: migrate twins undeclared store blobs (finalize-consistency, EGB-710)" -``` - ---- - -## Task 3: `secrets migrate --status` survey - -**Files:** -- Modify: `secrets` — add `_migrate_status()`; extend `cmd_migrate` (`secrets:2267-2285`) -- Test: `test/migrate.bats` - -- [ ] **Step 1: Write the failing tests** - -Add to `test/migrate.bats`: - -```bash -@test "migrate --status flags a project that needs migrating" { - make_v1_store - m_gradle_src $'beaconClerkPkTest=pk_test_abc\n' - create_project_dir needsmig - printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files - "$SECRETS_BIN" push needsmig >/dev/null 2>&1 # v1 blob, no twin yet - run "$SECRETS_BIN" migrate --status - [ "$status" -ne 0 ] # not finalize-ready - [[ "$output" == *"needsmig"* ]] || false - [[ "$output" == *"NEEDS MIGRATE"* ]] || false - [[ "$output" == *"Not finalize-ready"* ]] || false -} - -@test "migrate --status reports finalize-ready once every blob is twinned" { - make_v1_store - m_gradle_src $'beaconClerkPkTest=pk_test_abc\n' - create_project_dir readymig - printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files - "$SECRETS_BIN" push readymig >/dev/null 2>&1 - "$SECRETS_BIN" migrate >/dev/null 2>&1 # create the twin - run "$SECRETS_BIN" migrate --status - [ "$status" -eq 0 ] - [[ "$output" == *"Finalize-ready"* ]] || false -} - -@test "migrate --status on an already-v2 store says nothing to do" { - init_with_remote - create_project_dir v2status - run "$SECRETS_BIN" migrate --status - [ "$status" -eq 0 ] - [[ "$output" == *"v2"* ]] || false -} -``` - -- [ ] **Step 2: Run them to confirm they fail** - -Run: `bats test/migrate.bats -f "status"` -Expected: FAIL — `--status` is an unknown flag today (`die "Unknown migrate flag: --status…"`, status 1), so the assertions fail. - -- [ ] **Step 3: Add `_migrate_status` (place it just before `_migrate_finalize`, ~`secrets:2195`)** - -```bash -# Read-only survey: walk every project dir in the store and report each one's -# v2-readiness from the blobs on disk (no manifest, no decryption). Exits -# non-zero when any v1 properties blob lacks a v2 twin (i.e. the store is not -# yet finalize-ready) so it can gate scripting, mirroring `verify`'s posture. -_migrate_status() { - if [ "$(_store_format)" = "2" ]; then - info "Store format: v2 (finalized) — nothing to migrate." - return 0 - fi - echo "Store format: v1 (not finalized). Per-project migration status:" - local any_untwinned=0 dir project v1 untwinned f new - for dir in "$SECRETS_DIR"/*/; do - [ -d "$dir" ] || continue - project=$(basename "$dir") - case "$project" in .*) continue ;; esac - v1=0; untwinned=0 - while IFS= read -r f; do - [ -f "$f" ] || continue - v1=$((v1 + 1)) - new="${f%.gradle-properties.age}.properties.age" - [ -f "$new" ] || untwinned=$((untwinned + 1)) - done < <(find "$dir" -type f -name '*.gradle-properties.age' 2>/dev/null) - if [ "$v1" -eq 0 ]; then - echo " $project: v2-ready (no v1 properties blobs)" - elif [ "$untwinned" -eq 0 ]; then - echo " $project: migrated ($v1 v1 blob(s), all twinned)" - else - echo " $project: NEEDS MIGRATE ($untwinned of $v1 v1 blob(s) un-twinned) — cd into the project and run 'secrets migrate'" - any_untwinned=1 - fi - done - echo - if [ "$any_untwinned" -eq 1 ]; then - echo "Not finalize-ready: migrate the projects marked NEEDS MIGRATE, then run 'secrets migrate --finalize'." - return 1 - fi - echo "Finalize-ready: every v1 properties blob has a v2 twin. Run 'secrets migrate --finalize' once every machine is upgraded." - return 0 -} -``` - -- [ ] **Step 4: Wire `--status` into `cmd_migrate`** - -In `cmd_migrate()` (`secrets:2267`), add the `status` local and parse + dispatch. Replace: - -```bash - local dry_run=false finalize=false force=false - while [ $# -gt 0 ]; do - case "$1" in - --dry-run) dry_run=true; shift ;; - --finalize) finalize=true; shift ;; - --yes) force=true; shift ;; - -*) die "Unknown migrate flag: $1. Usage: secrets migrate [--dry-run | --finalize] [--yes]" ;; - *) die "migrate takes no project argument. Run it from inside a project (copy-forward) or use --finalize (store-wide)." ;; - esac - done - if [ "$finalize" = true ]; then - _migrate_finalize "$force" - else - _migrate_project "$dry_run" - fi -``` - -with: - -```bash - local dry_run=false finalize=false force=false status=false - while [ $# -gt 0 ]; do - case "$1" in - --dry-run) dry_run=true; shift ;; - --finalize) finalize=true; shift ;; - --status) status=true; shift ;; - --yes) force=true; shift ;; - -*) die "Unknown migrate flag: $1. Usage: secrets migrate [--dry-run | --status | --finalize] [--yes]" ;; - *) die "migrate takes no project argument. Run it from inside a project (copy-forward), or use --status / --finalize (store-wide)." ;; - esac - done - if [ "$status" = true ]; then - _migrate_status - elif [ "$finalize" = true ]; then - _migrate_finalize "$force" - else - _migrate_project "$dry_run" - fi -``` - -- [ ] **Step 5: Run the status tests + full suite** - -Run: `bats test/migrate.bats` -Expected: all PASS. - -- [ ] **Step 6: Commit** - -```bash -git add secrets test/migrate.bats -git commit -m "feat: secrets migrate --status surveys per-project v2 readiness (EGB-710)" -``` - ---- - -## Task 4: Docs, help text, version, changelog - -**Files:** -- Modify: `secrets` — `cmd_help` (`secrets:2308-2309`) -- Modify: `CLAUDE.md`, `README.md`, `VERSION`, `CHANGELOG.md` - -- [ ] **Step 1: Update `cmd_help` migrate lines** - -In `cmd_help()` replace the two migrate lines (`secrets:2308-2309`): - -``` - secrets migrate [--dry-run] Copy-forward this project's blobs to store format v2 - secrets migrate --finalize Drop v1 blobs and mark the store v2 (after verify) -``` - -with: - -``` - secrets migrate [--dry-run] Copy-forward this project's v1 blobs to store format v2 - secrets migrate --status Survey every project's v2 readiness (finalize gate) - secrets migrate --finalize Drop v1 blobs and mark the store v2 (after verify) -``` - -- [ ] **Step 2: Update `CLAUDE.md` migrate paragraph** - -In the "Store format (EGB-677 stage 2 / EGB-703)" bullet, update the migration description: copy-forward is now **manifest-free** — `secrets migrate` enumerates the project's `*.gradle-properties.age` store blobs directly (same source of truth as `--finalize`), so a legacy `.secrets-files`-only project migrates without a `.secrets.json` and no store blob is left un-twinned. Add `secrets migrate --status` to the workflow line as the read-only survey that reports per-project readiness and gates `--finalize`. Reference EGB-710. - -- [ ] **Step 3: Update `README.md`** - -Find the migrate section/help block and add the `--status` line and the manifest-free note, mirroring the help text. - -- [ ] **Step 4: Bump `VERSION`** - -Set `VERSION` to `0.6.1.0`. - -- [ ] **Step 5: Add `CHANGELOG.md` entry** - -Insert above `## [0.6.0.1]`: - -```markdown -## [0.6.1.0] - 2026-06-08 - -### Changed - -- **`secrets migrate` copy-forward is now manifest-free (EGB-710)** — the - per-project step enumerates the store's `*.gradle-properties.age` blobs - directly (the same source of truth `--finalize` uses) instead of reading - `.secrets.json`. A legacy `.secrets-files`-only project now migrates cleanly - instead of dead-ending with "No .secrets.json", and a store blob the manifest - no longer declares still gets a v2 twin (so `--finalize` won't refuse it). - Running migrate in a project with no v1 properties blobs is a clean no-op. - -### Added - -- **`secrets migrate --status`** — a read-only survey that walks every project - in the store and reports its v2 readiness (v2-ready / migrated / NEEDS - MIGRATE), then whether the store as a whole is finalize-ready. Exits non-zero - while any v1 blob is un-twinned, so it can gate the path to `--finalize`. -``` - -- [ ] **Step 6: Run the full bats suite (all files)** - -Run: `bats test/` -Expected: all PASS (secrets.bats + manifest.bats + migrate.bats). Confirm the count went up by the tests added here. - -- [ ] **Step 7: Commit** - -```bash -git add secrets CLAUDE.md README.md VERSION CHANGELOG.md -git commit -m "docs: migrate --status + manifest-free copy-forward; bump 0.6.1.0 (EGB-710)" -``` - ---- - -## Self-review against the EGB-710 spec - -- **AC: dotenv-only project exits 0 "already v2-shaped"** → Task 1 (`moved==0 && already==0` branch, substring `no v1 properties blobs`). Test: "no manifest and no store blobs is a clean no-op" (Task 1 Step 5). -- **AC: `.secrets-files`-only project with v1 blobs migrates instead of the generic error** → the manifest-free rewrite makes it *just migrate* (better than guiding to push first). Test: "manifest-free" (Task 1 Step 1). Note the rewrite supersedes the ticket's "offer to run absorb then migrate" branch — migrate no longer needs the manifest, so there's nothing to prompt for; the only destructive step (`--finalize`) keeps its existing `/dev/tty` confirmation. -- **AC: interactive prompts read `/dev/tty`, degrade non-interactively, bash 3.2** → no new prompt is introduced (copy-forward is non-destructive); `--finalize`'s existing `/dev/tty` confirm is untouched. `--status` is pure read-only output. All new code is bash 3.2 (no associative arrays, `[[ ]]` only in tests with `|| false`). -- **AC: no behavior change to copy-forward/finalize safety gates** → finalize untouched; copy-forward stays non-destructive (v1 kept, commit + push twins). Verified by the unchanged finalize tests (migrate.bats:168-214). -- **AC: bats coverage per branch** → Tasks 1-3 add: manifest-free migrate, no-op no-blobs, undeclared-blob twinning, `--status` needs-migrate / finalize-ready / already-v2. -- **Stretch: `--status` survey** → Task 3, delivered. -- **Sequencing: independent of the EGB-709 gate** → confirmed; this only touches migrate ergonomics and helps operators *reach* all-v2. EGB-703 safety posture (recovery tag, verify-green gate, twin-before-drop) is preserved. - -## Operator-local follow-up (not part of this plan) - -This repo's `.ship-policy.json` opts out of AI security/red-team passes; before any PR, ask the operator to run `./test/run-security.sh` and complete the SIGNOFF. After landing, the operator can re-run their real-store migration for `onefinalmessage` et al. (`secrets migrate` per project → `secrets migrate --finalize`), which is the path to clearing the EGB-709 v2 gate. diff --git a/docs/superpowers/plans/2026-06-08-egb-713-version-skew-nudge.md b/docs/superpowers/plans/2026-06-08-egb-713-version-skew-nudge.md deleted file mode 100644 index 61d1ce5..0000000 --- a/docs/superpowers/plans/2026-06-08-egb-713-version-skew-nudge.md +++ /dev/null @@ -1,300 +0,0 @@ -# EGB-713: Version-skew nudge Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: superpowers:executing-plans / subagent-driven-development. Steps use `- [ ]`. - -**Goal:** Warn (non-fatally) when the active store was last written by a newer `secrets` version than the running client, so a behind user is told to update — the loud counterpart to EGB-712's quiet forcing function. - -**Architecture:** Stamp the store with the highest writer `VERSION` seen (`$SECRETS_DIR/.secrets-writer-version`, committed, monotonic) on every store-committing write. On any store command, compare that stamp to the client's own `VERSION` (read from `$SCRIPT_DIR/VERSION`); if the stamp is newer, print a one-time stderr nudge. Legacy stores with no stamp are silent. - -**Tech Stack:** bash 3.2 (`secrets`); bats. Spec/idea: ticket **EGB-713**. - -## Background (verified against `main`, post-EGB-712) -- `SCRIPT_DIR` already defined at `secrets:21` (`$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)`). The repo-root `VERSION` file lives next to the script. -- Store-committing write sites (each does `git -C "$SECRETS_DIR" add -A`): `commit_and_push_secrets()` `secrets:1265` (push), rekey `secrets:1822`, migrate copy-forward `secrets:2215`, finalize no-v1 `secrets:2305`, finalize drop `secrets:2340`. Plus `cmd_init` (born store) and `cmd_rm`. -- `ensure_store_protections()` (`secrets:1144`) is shared with pull (read) — do NOT stamp there. -- `check_initialized()` (`secrets:54`) early-returns when `$SECRETS_DIR/.git` exists and is called by push/pull/list/rm/rekey/verify/migrate — the natural warning hook. -- `cmd_which()` (`secrets:1945`) prints `format: v$(_store_format)` — add the writer-version line here. -- `.gitignore` only ignores `key.txt`, so `.secrets-writer-version` commits normally. -- VERSION currently `0.7.0.0` → bump to `0.7.1.0`. -- Baseline: `bats test/` = 246 passing. New tests land in a new file `test/version.bats`. - ---- - -## Task 1: Version helpers + comparator (TDD) - -**Files:** `secrets`, `test/version.bats` (new). - -- [ ] Step 1: Create `test/version.bats` testing the comparator via a tiny harness that sources the script's functions is awkward (the script runs main). Instead test through observable behavior in later tasks; for the comparator, add a hidden debug subcommand is overkill. Use this approach: test `_version_gt` indirectly by exporting it is not possible. So test the comparator by adding the helpers and a **`secrets __vercmp `** internal is overkill too. Decision: test the comparator's *effect* in Task 3 (warning) and Task 2 (stamp monotonicity), which exercise it end-to-end. For Task 1, write the helpers and verify with a one-off `bash -c` sourcing guard. - -Add to `test/version.bats`: -```bash -load test_helper - -# Exercises the comparator through a bash subshell that defines the same logic -# the script uses, guarding the numeric (not lexical) ordering contract. -@test "version comparator orders 0.7.0.0 < 0.10.0.0 numerically" { - run bash -c ' - _version_gt() { - local a="$1" b="$2" i ai bi; local -a af bf - IFS=. read -r -a af <<< "$a"; IFS=. read -r -a bf <<< "$b" - for i in 0 1 2 3; do - ai=${af[$i]:-0}; ai=${ai//[!0-9]/}; [ -n "$ai" ] || ai=0 - bi=${bf[$i]:-0}; bi=${bi//[!0-9]/}; [ -n "$bi" ] || bi=0 - if [ "$((10#$ai))" -gt "$((10#$bi))" ]; then return 0; fi - if [ "$((10#$ai))" -lt "$((10#$bi))" ]; then return 1; fi - done; return 1 - } - _version_gt 0.10.0.0 0.7.0.0 && echo "10gt7" - _version_gt 0.7.0.0 0.10.0.0 || echo "7not_gt_10" - _version_gt 0.7.0.0 0.7.0.0 || echo "equal_not_gt" - ' - [ "$status" -eq 0 ] - [[ "$output" == *"10gt7"* ]] || false - [[ "$output" == *"7not_gt_10"* ]] || false - [[ "$output" == *"equal_not_gt"* ]] || false -} -``` - -- [ ] Step 2: Run `bats test/version.bats` → PASS (pins the contract the script must match). - -- [ ] Step 3: Add the helpers to `secrets` (near `_store_format`, after `SCRIPT_DIR`/version constants — place after the `MANIFEST_VERSION=2` area or near `_store_format`): -```bash -# The running client's own version, read from the VERSION file shipped beside -# the script. Empty/"0.0.0.0" if absent (e.g. an odd install) — treated as -# "unknown/oldest" so a missing VERSION never triggers a spurious nudge. -_client_version() { - local v="" - [ -f "$SCRIPT_DIR/VERSION" ] && v=$(head -1 "$SCRIPT_DIR/VERSION" 2>/dev/null | tr -d '\r\n[:space:]') - printf '%s' "${v:-0.0.0.0}" -} - -# Numeric four-field (MAJOR.MINOR.PATCH.MICRO) compare. Returns 0 iff $1 > $2. -# Per-field numeric (so 0.10.0.0 > 0.7.0.0); missing/garbage fields → 0. -_version_gt() { - local a="$1" b="$2" i ai bi; local -a af bf - IFS=. read -r -a af <<< "$a"; IFS=. read -r -a bf <<< "$b" - for i in 0 1 2 3; do - ai=${af[$i]:-0}; ai=${ai//[!0-9]/}; [ -n "$ai" ] || ai=0 - bi=${bf[$i]:-0}; bi=${bi//[!0-9]/}; [ -n "$bi" ] || bi=0 - if [ "$((10#$ai))" -gt "$((10#$bi))" ]; then return 0; fi - if [ "$((10#$ai))" -lt "$((10#$bi))" ]; then return 1; fi - done - return 1 -} - -WRITER_VERSION_FILE_NAME=".secrets-writer-version" -# Highest client version recorded as having written to the store (empty if the -# store predates this feature — "minus the initial builds", silent by design). -_store_writer_version() { - local f="$SECRETS_DIR/$WRITER_VERSION_FILE_NAME" - [ -f "$f" ] && head -1 "$f" 2>/dev/null | tr -d '\r\n[:space:]' -} -``` - -- [ ] Step 4: `bash -n secrets` parses; `bats test/` still 246 + 1 (the comparator test) = 247. -- [ ] Step 5: Commit: `git add secrets test/version.bats && git commit -m "feat: version helpers + numeric comparator (EGB-713)"` - ---- - -## Task 2: Stamp the writer-version on write (TDD) - -**Files:** `secrets`, `test/version.bats`. - -- [ ] Step 1: Add tests: -```bash -@test "push stamps the store writer-version with the client version" { - init_with_remote - create_project_dir wvstamp - "$SECRETS_BIN" push wvstamp >/dev/null 2>&1 - [ -f "$SECRETS_DIR/.secrets-writer-version" ] - run cat "$SECRETS_DIR/.secrets-writer-version" - [ "$output" = "$(cat "$(dirname "$SECRETS_BIN")/VERSION")" ] -} - -@test "writer-version stamp is monotonic (a push never lowers a higher stamp)" { - init_with_remote - create_project_dir wvmono - printf '9.9.9.9\n' > "$SECRETS_DIR/.secrets-writer-version" - "$SECRETS_BIN" push wvmono >/dev/null 2>&1 - run cat "$SECRETS_DIR/.secrets-writer-version" - [ "$output" = "9.9.9.9" ] # not lowered to the client's version -} - -@test "writer-version stamp is committed, not gitignored" { - init_with_remote - create_project_dir wvcommit - "$SECRETS_BIN" push wvcommit >/dev/null 2>&1 - run bash -c "git -C $SECRETS_DIR ls-files | grep -qx .secrets-writer-version" - [ "$status" -eq 0 ] -} -``` - -- [ ] Step 2: Run `bats test/version.bats -f "stamp"` → FAIL (no stamping yet). - -- [ ] Step 3: Add the stamp helper (after `_store_writer_version`): -```bash -# Raise the store's recorded writer-version to the client's version (monotonic; -# never lowers it). Called right before each store-committing `git add -A` so -# the stamp rides the same commit. Read paths (pull) never call this. -_stamp_writer_version() { - local cur cli - cur=$(_store_writer_version) - cli=$(_client_version) - if [ -z "$cur" ] || _version_gt "$cli" "$cur"; then - printf '%s\n' "$cli" > "$SECRETS_DIR/$WRITER_VERSION_FILE_NAME" - fi -} -``` - -- [ ] Step 4: Call `_stamp_writer_version` immediately before each store-committing `git -C "$SECRETS_DIR" add -A`: - - `secrets:1265` (in `commit_and_push_secrets`, before `git add -A`) - - `secrets:1822` (rekey) - - `secrets:2215` (migrate copy-forward) - - `secrets:2305` (finalize, no-v1 path) - - `secrets:2340` (finalize, drop path) - Also in `cmd_init`, after the store repo is created and before its first commit (so a born store records its version), and in `cmd_rm` before its commit. - Each insertion is the single line ` _stamp_writer_version` at the matching indentation directly above the `git ... add -A` (or before the `git ... commit` where there's no add -A, e.g. rm/init — there, stamp then ensure it's staged via the existing add/commit). - -- [ ] Step 5: `bats test/version.bats` → all pass. `bats test/` → 250 (247 + 3). -- [ ] Step 6: Commit: `git add secrets test/version.bats && git commit -m "feat: stamp store writer-version on write, monotonic (EGB-713)"` - ---- - -## Task 3: Skew warning on command (TDD) - -**Files:** `secrets`, `test/version.bats`. - -- [ ] Step 1: Add tests: -```bash -@test "a store written by a newer version warns on a command (non-fatal)" { - init_with_remote - create_project_dir skewwarn - "$SECRETS_BIN" push skewwarn >/dev/null 2>&1 - printf '99.0.0.0\n' > "$SECRETS_DIR/.secrets-writer-version" - run "$SECRETS_BIN" list - [ "$status" -eq 0 ] # non-fatal - [[ "$output" == *"newer"* || "$output" == *"update"* ]] || false -} - -@test "a store at the same/older version is silent" { - init_with_remote - create_project_dir noskew - "$SECRETS_BIN" push noskew >/dev/null 2>&1 # stamp == client version - run "$SECRETS_BIN" list - [ "$status" -eq 0 ] - [[ "$output" != *"update your secrets"* ]] || false -} - -@test "a store with no writer-version marker is silent (legacy store)" { - init_with_remote - create_project_dir legacynostamp - "$SECRETS_BIN" push legacynostamp >/dev/null 2>&1 - rm -f "$SECRETS_DIR/.secrets-writer-version" - run "$SECRETS_BIN" list - [ "$status" -eq 0 ] - [[ "$output" != *"update your secrets"* ]] || false -} -``` - -- [ ] Step 2: Run `bats test/version.bats -f "skew\|silent\|legacy"` → the "newer" test FAILS (no warning yet). - -- [ ] Step 3: Add the skew check (after `_stamp_writer_version`): -```bash -# Warn ONCE per invocation if the store was last written by a newer client than -# us. Non-fatal (read/write paths keep their exit codes). Silent when the store -# carries no writer-version (legacy) or is same/older than us. -_VERSION_SKEW_WARNED=0 -_check_store_version_skew() { - [ "$_VERSION_SKEW_WARNED" = 1 ] && return 0 - local sv cv - sv=$(_store_writer_version) - [ -n "$sv" ] || return 0 - cv=$(_client_version) - if _version_gt "$sv" "$cv"; then - _VERSION_SKEW_WARNED=1 - echo "NOTE: this store was last written by secrets v$sv; you're on v$cv." >&2 - echo " Update your secrets tool: git -C \"$SCRIPT_DIR\" pull" >&2 - fi - return 0 -} -``` - -- [ ] Step 4: Hook it into `check_initialized` — change `secrets:55-57`: -```bash - if [ -d "$SECRETS_DIR/.git" ]; then - return - fi -``` -to: -```bash - if [ -d "$SECRETS_DIR/.git" ]; then - _check_store_version_skew - return - fi -``` - -- [ ] Step 5: `bats test/version.bats` → all pass. `bats test/` → 253. -- [ ] Step 6: Commit: `git add secrets test/version.bats && git commit -m "feat: warn on store version skew (once per invocation, EGB-713)"` - ---- - -## Task 4: `secrets which` surfaces the writer-version (TDD) - -**Files:** `secrets`, `test/version.bats`. - -- [ ] Step 1: Add test: -```bash -@test "which prints the store writer-version and a behind note" { - init_with_remote - create_project_dir whichwv - "$SECRETS_BIN" push whichwv >/dev/null 2>&1 - printf '99.0.0.0\n' > "$SECRETS_DIR/.secrets-writer-version" - run "$SECRETS_BIN" which - [ "$status" -eq 0 ] - [[ "$output" == *"written-by: v99.0.0.0"* ]] || false - [[ "$output" == *"behind"* || "$output" == *"update"* ]] || false -} -``` - -- [ ] Step 2: Run → FAIL (which doesn't print written-by). - -- [ ] Step 3: In `cmd_which`, after the `echo "format: v$(_store_format)"` line (`secrets:1951`), add: -```bash - local _wv; _wv=$(_store_writer_version) - if [ -n "$_wv" ]; then - local _cv; _cv=$(_client_version) - if _version_gt "$_wv" "$_cv"; then - echo "written-by: v$_wv (you're on v$_cv — behind; run: git -C \"$SCRIPT_DIR\" pull)" - else - echo "written-by: v$_wv" - fi - fi -``` -Note: `cmd_which` calls `resolve_store` but may not call `check_initialized`, so this also avoids double-printing the skew NOTE; the `which` line is the dedicated surface. - -- [ ] Step 4: `bats test/version.bats` → pass. `bats test/` → 254. -- [ ] Step 5: Commit: `git add secrets test/version.bats && git commit -m "feat: secrets which shows store writer-version + behind note (EGB-713)"` - ---- - -## Task 5: Docs + version bump - -**Files:** `secrets` (cmd_help unchanged unless adding a note), `CLAUDE.md`, `README.md`, `VERSION`, `CHANGELOG.md`. - -- [ ] Step 1: `CLAUDE.md` — add to the "Store format" bullet a sentence on the writer-version: a committed `.secrets-writer-version` records the highest client `VERSION` that has written (monotonic, stamped on store-committing writes); commands warn once (stderr, non-fatal) when the store's stamp exceeds the running client, and `secrets which` shows `written-by: vN`. Legacy stores (no marker) are silent. (EGB-713.) -- [ ] Step 2: `README.md` — under the upgrading section, note that an out-of-date `secrets` prints a one-line "update" nudge when it touches a store newer than itself. -- [ ] Step 3: `VERSION` → `0.7.1.0`. -- [ ] Step 4: `CHANGELOG.md` — new `## [0.7.1.0] - 2026-06-08` with an Added entry for the version-skew nudge + `secrets which` writer-version line. -- [ ] Step 5: `bats test/` → all green (254). `./secrets which` against a scratch store renders (covered by tests). -- [ ] Step 6: Commit: `git add secrets CLAUDE.md README.md VERSION CHANGELOG.md && git commit -m "docs: version-skew nudge + writer-version; bump 0.7.1.0 (EGB-713)"` - ---- - -## Self-review vs ticket AC -- "Newer stamp → nudge; same/older → silent" → Task 3. -- "No marker → silent (legacy)" → Task 3 + `_store_writer_version` empty. -- "Monotonic, committed" → Task 2. -- "Numeric comparator (0.7.0.0 < 0.10.0.0)" → Task 1. -- "Non-fatal, never changes read/pull exit codes" → Task 3 (`_check_store_version_skew` always `return 0`). -- "`which` surfaces it" → Task 4. "Warn once per invocation" → `_VERSION_SKEW_WARNED` guard. -- bash 3.2: `read -a`, `local -a`, `10#`, parameter strips — all 3.2-safe. diff --git a/docs/superpowers/plans/2026-06-24-multi-recipient-age-encryption.md b/docs/superpowers/plans/2026-06-24-multi-recipient-age-encryption.md deleted file mode 100644 index 54a38d8..0000000 --- a/docs/superpowers/plans/2026-06-24-multi-recipient-age-encryption.md +++ /dev/null @@ -1,1207 +0,0 @@ -# Multi-recipient age encryption Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Let a single `secrets` store encrypt every blob to N age recipient public keys (a per-store team-key set) instead of one shared key, managed via `secrets recipients add/rm/list`, with full backward compatibility for existing single-key stores. - -**Architecture:** A committed, store-scoped `recipients.txt` (age `-R` format) holds the recipient set. A new `_load_recipients` parses + validates it into a global `RECIPIENT_ARGS=(-r k1 -r k2 …)` array that every encrypt site uses; absence of the file means legacy single-key behavior (`-r $(get_pubkey)`). A shared `_reencrypt_all` engine (factored from today's `rekey`) decrypts the whole store with the local key and re-encrypts to the current set, and is called by `recipients add/rm`, the new `reencrypt`, and multi-recipient `rekey`. Decryption is unchanged — each member uses their own `key.txt`. - -**Tech Stack:** Single POSIX-ish bash script (`secrets`), system bash 3.2 compatible (indexed arrays OK, NO associative arrays). `age` / `age-keygen` for crypto. `git` for the store. `bats-core` for tests. `jq` is NOT introduced anywhere in this feature (recipients.txt is plain text, keeping store ops jq-free). - -## Global Constraints - -- **bash 3.2 only:** no associative arrays, no bash-4 features. Indexed arrays (`RECIPIENT_ARGS=()`, `arr+=(x)`, `"${arr[@]}"`) are fine. -- **bats `[[ ]]` gotcha:** every standalone `[[ … ]]` assertion in a test MUST end with `|| false`. Single-bracket `[ ]` is unaffected. -- **age recipient format accepted:** native age X25519 only — `age1` + exactly 58 chars of `[0-9a-z]`. SSH recipients are out of scope (reject them). This regex/charset is also the injection rail: it cannot contain shell metacharacters, whitespace, or extra flags. -- **`recipients.txt` is committed, NOT gitignored** (public keys are not secret). The store `.gitignore` only blocks `key.txt` and plaintext env files, so the file is tracked automatically — do not add it to `.gitignore`. -- **Security-review policy (`.ship-policy.json`, CLAUDE.md):** adversarial fixtures in this plan are ordinary bats regression tests, NOT AI red-team passes. Do NOT run `./test/run-security.sh` on the user's behalf. Before ship, the human operator runs it and signs off. -- **Re-encrypt invariant:** any change to the recipient set re-encrypts the WHOLE store in one commit. `RECIPIENT_ARGS` is always populated by `_load_recipients` before any `age "${RECIPIENT_ARGS[@]}"` call (never reference the array empty under `set -u`). -- **Commit cadence:** one commit per task (TDD: test → impl → green → commit). - -## File map - -- `secrets` — all code changes (helpers, `recipients`/`reencrypt` commands, encrypt-site refactor, `init`/`which`/`verify`/`rekey` edits, dispatch + help). -- `test/recipients.bats` — NEW suite for all multi-recipient behavior + security fixtures. -- `CLAUDE.md`, `README.md` — docs + test counts. - -## Conventions referenced - -- Constants like `SECRETS_FILES_NAME=".secrets-files"` live ~line 360; `KEY_FILE` is set both as a global default (~line 20) and re-set inside `resolve_store` (~line 301). Mirror this for `RECIPIENTS_FILE`. -- Existing encrypt sites (all `age -r "$pubkey" -o …`): `push_dir_to_project` (~1207), `cmd_push` inline (~1357), `push_external_files` (~655 and ~684), `cmd_rekey` (~1778). `get_pubkey` (~98) derives the pubkey from `key.txt`. -- Tests run via `run "$SECRETS_BIN" ` with isolated `$HOME` and `$SECRETS_DIR`; helpers `init_with_remote`, `create_project_dir` live in `test/test_helper.bash`. - ---- - -### Task 1: Recipient core (`RECIPIENTS_FILE`, validation, `_load_recipients`) wired into the push encrypt path - -**Files:** -- Modify: `secrets` (constants ~line 19-21; `resolve_store` ~301; new helpers after `get_pubkey` ~99; encrypt sites ~655, ~684, ~1207, ~1357; `cmd_push` ~1268; `cmd_push_workspaces` ~1428; `push_dir_to_project`/`push_external_files` signatures) -- Test: `test/recipients.bats` (new) - -**Interfaces:** -- Produces: global `RECIPIENT_ARGS` (indexed array of `-r ` pairs); `RECIPIENTS_FILE` / `RECIPIENTS_FILE_NAME`; `_validate_age_recipient ` (0 = valid age1 key); `_load_recipients` (populates `RECIPIENT_ARGS`, dies on bad/symlinked/empty file). -- Consumes: `get_pubkey`, `SECRETS_DIR`, `KEY_FILE`. - -- [ ] **Step 1: Write failing tests** in new `test/recipients.bats`: - -```bash -#!/usr/bin/env bats -load test_helper - -# A throwaway second identity for "another teammate". -make_second_identity() { - age-keygen -o "$TEST_TMPDIR/bob.txt" 2>/dev/null - BOB_PUB=$(age-keygen -y "$TEST_TMPDIR/bob.txt") -} - -@test "push without recipients.txt stays single-key (legacy behavior)" { - init_with_remote - create_project_dir myproj - run "$SECRETS_BIN" push - [ "$status" -eq 0 ] - # No recipients.txt was created by push. - [ ! -e "$SECRETS_DIR/recipients.txt" ] - # Blob decrypts with the store's own key. - run age -d -i "$SECRETS_DIR/key.txt" "$SECRETS_DIR/myproj/.env.age" - [ "$status" -eq 0 ] -} - -@test "push with a hand-written recipients.txt encrypts to every listed key" { - init_with_remote - make_second_identity - STORE_PUB=$(age-keygen -y "$SECRETS_DIR/key.txt") - printf '# self\n%s\n# bob\n%s\n' "$STORE_PUB" "$BOB_PUB" > "$SECRETS_DIR/recipients.txt" - create_project_dir myproj - run "$SECRETS_BIN" push - [ "$status" -eq 0 ] - # Bob (a recipient) can decrypt the pushed blob with HIS key. - run age -d -i "$TEST_TMPDIR/bob.txt" "$SECRETS_DIR/myproj/.env.age" - [ "$status" -eq 0 ] - # And the store key still can too. - run age -d -i "$SECRETS_DIR/key.txt" "$SECRETS_DIR/myproj/.env.age" - [ "$status" -eq 0 ] -} - -@test "push refuses a recipients.txt with an invalid key" { - init_with_remote - STORE_PUB=$(age-keygen -y "$SECRETS_DIR/key.txt") - printf '%s\nnot-an-age-key\n' "$STORE_PUB" > "$SECRETS_DIR/recipients.txt" - create_project_dir myproj - run "$SECRETS_BIN" push - [ "$status" -ne 0 ] - [[ "$output" == *"Invalid recipient"* ]] || false -} - -@test "push refuses a symlinked recipients.txt" { - init_with_remote - STORE_PUB=$(age-keygen -y "$SECRETS_DIR/key.txt") - printf '%s\n' "$STORE_PUB" > "$TEST_TMPDIR/elsewhere.txt" - ln -s "$TEST_TMPDIR/elsewhere.txt" "$SECRETS_DIR/recipients.txt" - create_project_dir myproj - run "$SECRETS_BIN" push - [ "$status" -ne 0 ] - [[ "$output" == *"symlink"* ]] || false -} -``` - -- [ ] **Step 2: Run to verify they fail** - -Run: `bats test/recipients.bats` -Expected: FAIL (recipients.txt is ignored today; multi-recipient blob won't decrypt with bob's key; invalid/symlink cases don't error). - -- [ ] **Step 3: Add the constant + `RECIPIENTS_FILE` plumbing** - -Near `KEY_FILE="$SECRETS_DIR/key.txt"` (~line 20), add the name constant just above it and the path just below: - -```bash -RECIPIENTS_FILE_NAME="recipients.txt" -KEY_FILE="$SECRETS_DIR/key.txt" -RECIPIENTS_FILE="$SECRETS_DIR/$RECIPIENTS_FILE_NAME" -``` - -Inside `resolve_store`, right after the line that re-sets `KEY_FILE="$SECRETS_DIR/key.txt"` (~line 301), add: - -```bash - RECIPIENTS_FILE="$SECRETS_DIR/$RECIPIENTS_FILE_NAME" -``` - -- [ ] **Step 4: Add `_validate_age_recipient` and `_load_recipients`** immediately after `get_pubkey` (~line 99): - -```bash -# A native age X25519 recipient: "age1" + exactly 58 chars of [0-9a-z]. -# This is also the injection rail — it cannot hold shell metacharacters, -# whitespace, control chars, or extra flags. SSH recipients are intentionally -# unsupported (EGB-283 scope cut). -_validate_age_recipient() { - case "$1" in - age1*) : ;; - *) return 1 ;; - esac - local body="${1#age1}" - [ "${#body}" -eq 58 ] || return 1 - case "$body" in - *[!0-9a-z]*) return 1 ;; - esac - return 0 -} - -# Populate the global RECIPIENT_ARGS array with one "-r " per store -# recipient. recipients.txt present -> validated keys from the file (the store -# is multi-recipient). Absent -> the single pubkey derived from key.txt (legacy -# single-key store, exactly today's behavior). We parse the file ourselves -# (never `age -R `) because it is committed = an injection surface; every -# line is validated and the file is refused if symlinked. Dies on any problem. -RECIPIENT_ARGS=() -_load_recipients() { - RECIPIENT_ARGS=() - if [ ! -e "$RECIPIENTS_FILE" ]; then - RECIPIENT_ARGS=(-r "$(get_pubkey)") - return 0 - fi - if [ -L "$RECIPIENTS_FILE" ]; then - die "Refusing to read symlinked $RECIPIENTS_FILE_NAME (security)." - fi - local line trimmed n=0 - while IFS= read -r line || [ -n "$line" ]; do - trimmed="${line#"${line%%[![:space:]]*}"}" # lstrip - trimmed="${trimmed%"${trimmed##*[![:space:]]}"}" # rstrip - [ -z "$trimmed" ] && continue - case "$trimmed" in '#'*) continue ;; esac - if ! _validate_age_recipient "$trimmed"; then - die "Invalid recipient in $RECIPIENTS_FILE_NAME: '$trimmed' (expected a native age key: age1...)." - fi - RECIPIENT_ARGS+=(-r "$trimmed") - n=$((n + 1)) - done < "$RECIPIENTS_FILE" - if [ "$n" -eq 0 ]; then - die "$RECIPIENTS_FILE_NAME has no recipients — a store must have at least one. Run 'secrets recipients add '." - fi -} -``` - -- [ ] **Step 5: Route every encrypt site through `RECIPIENT_ARGS`** - -In `push_dir_to_project`, change the encrypt line (~1207): -```bash - age "${RECIPIENT_ARGS[@]}" -o "$SECRETS_DIR/$project/${name}.age" "$f" -``` -and delete its now-unused `local pubkey="$3"` line (~1192). - -In `cmd_push`, change the inline encrypt (~1357): -```bash - age "${RECIPIENT_ARGS[@]}" -o "$SECRETS_DIR/$project/${rel}.age" "$PWD/$rel" -``` - -In `push_external_files`, change both encrypt lines (~655 and ~684) to `age "${RECIPIENT_ARGS[@]}" -o …` (keep the rest of each line identical) and delete its `local pubkey="$3"` from the signature line `local root="$1" project="$2" pubkey="$3"` → `local root="$1" project="$2"`. - -- [ ] **Step 6: Load recipients in the push commands and drop the old `pubkey` threading** - -In `cmd_push` (~1268-1269) replace: -```bash - local pubkey - pubkey=$(get_pubkey) -``` -with: -```bash - _load_recipients -``` -and change the external call (~1365) `push_external_files "$PWD" "$project"` (drop `"$pubkey"`). - -In `cmd_push_workspaces` (~1428) replace the `pubkey=$(get_pubkey)` pair with `_load_recipients`, and drop the `"$pubkey"` argument from the `push_dir_to_project …` (~1432, ~1443) and `push_external_files …` (~1450) calls. - -- [ ] **Step 7: Run the tests** - -Run: `bats test/recipients.bats` -Expected: PASS (4 tests). - -- [ ] **Step 8: Run the full suite to confirm no regression** - -Run: `bats test/` -Expected: PASS (all existing tests still green — legacy push/pull unchanged). - -- [ ] **Step 9: Commit** - -```bash -git add secrets test/recipients.bats -git commit -m "feat: multi-recipient encrypt core + recipients.txt (EGB-283)" -``` - ---- - -### Task 2: `secrets recipients list` - -**Files:** -- Modify: `secrets` (new `_recipients_dump`, `cmd_recipients`, `_recipients_list`; dispatch + nothing in help yet) -- Test: `test/recipients.bats` - -**Interfaces:** -- Produces: `_recipients_dump` (emits `\t` per recipient, name = nearest preceding `# ` comment or empty); `cmd_recipients …` (routes `list`); `_recipients_list`. -- Consumes: `_load_recipients`, `RECIPIENTS_FILE`, `get_pubkey`. - -- [ ] **Step 1: Write failing tests** - -```bash -@test "recipients list on a legacy store shows the single derived key" { - init_with_remote - run "$SECRETS_BIN" recipients list - [ "$status" -eq 0 ] - [[ "$output" == *"single-key"* ]] || false - STORE_PUB=$(age-keygen -y "$SECRETS_DIR/key.txt") - [[ "$output" == *"$STORE_PUB"* ]] || false -} - -@test "recipients list shows names and keys from recipients.txt" { - init_with_remote - make_second_identity - STORE_PUB=$(age-keygen -y "$SECRETS_DIR/key.txt") - printf '# alice\n%s\n# bob\n%s\n' "$STORE_PUB" "$BOB_PUB" > "$SECRETS_DIR/recipients.txt" - run "$SECRETS_BIN" recipients list - [ "$status" -eq 0 ] - [[ "$output" == *"recipients: 2"* ]] || false - [[ "$output" == *"alice"* ]] || false - [[ "$output" == *"bob"* ]] || false -} -``` - -- [ ] **Step 2: Run to verify they fail** - -Run: `bats test/recipients.bats -f "recipients list"` -Expected: FAIL ("Unknown command: recipients"). - -- [ ] **Step 3: Add `_recipients_dump`** (place after `_load_recipients`): - -```bash -# Emit "\t" for each recipient in recipients.txt. is the most -# recent preceding "# " comment, or empty. Read-only; no validation -# (callers that need rails call _load_recipients separately). -_recipients_dump() { - [ -e "$RECIPIENTS_FILE" ] || return 0 - local line trimmed name="" - while IFS= read -r line || [ -n "$line" ]; do - trimmed="${line#"${line%%[![:space:]]*}"}" - trimmed="${trimmed%"${trimmed##*[![:space:]]}"}" - [ -z "$trimmed" ] && continue - case "$trimmed" in - '#'*) - name="${trimmed#\#}" - name="${name#"${name%%[![:space:]]*}"}" - ;; - *) - printf '%s\t%s\n' "$trimmed" "$name" - name="" - ;; - esac - done < "$RECIPIENTS_FILE" -} -``` - -- [ ] **Step 4: Add `cmd_recipients` + `_recipients_list`** (place near `cmd_which`): - -```bash -cmd_recipients() { - resolve_store - local sub="${1:-list}" - [ $# -gt 0 ] && shift - case "$sub" in - list) _recipients_list ;; - *) die "Unknown recipients subcommand: '$sub'. Usage: secrets recipients [list]" ;; - esac -} - -_recipients_list() { - check_initialized - if [ ! -e "$RECIPIENTS_FILE" ]; then - check_key - echo "recipients: single-key (no $RECIPIENTS_FILE_NAME)" - echo " $(get_pubkey)" - return 0 - fi - _load_recipients # validates the file (dies on bad key / symlink) - local count=0 k n - while IFS=$'\t' read -r k n; do count=$((count + 1)); done < <(_recipients_dump) - echo "recipients: $count (from $RECIPIENTS_FILE_NAME)" - while IFS=$'\t' read -r k n; do - if [ -n "$n" ]; then echo " $k ($n)"; else echo " $k"; fi - done < <(_recipients_dump) -} -``` - -- [ ] **Step 5: Wire dispatch.** In the `case "${1:-help}"` block, add above `which|where|status`: -```bash - recipients) shift; cmd_recipients "$@" ;; -``` - -- [ ] **Step 6: Run the tests** - -Run: `bats test/recipients.bats -f "recipients list"` -Expected: PASS. - -- [ ] **Step 7: Commit** - -```bash -git add secrets test/recipients.bats -git commit -m "feat: secrets recipients list (EGB-283)" -``` - ---- - -### Task 3: Shared `_reencrypt_all` engine + `secrets reencrypt` + dual `rekey` - -**Files:** -- Modify: `secrets` (new `_reencrypt_all`, `cmd_reencrypt`; rewrite `cmd_rekey` head to branch; dispatch) -- Test: `test/recipients.bats` - -**Interfaces:** -- Produces: `_reencrypt_all ` (decrypt whole store with `KEY_FILE`, re-encrypt to `RECIPIENT_ARGS`, commit + push; aborts with store intact on decrypt failure; no-op on empty store); `cmd_reencrypt`. -- Consumes: `_load_recipients`, `RECIPIENT_ARGS`, `KEY_FILE`, `ensure_store_protections`. - -- [ ] **Step 1: Write failing tests** - -```bash -@test "reencrypt re-encrypts existing blobs to a newly added recipient line" { - init_with_remote - create_project_dir myproj - run "$SECRETS_BIN" push # single-key blob (project name = "myproj"; blob at $SECRETS_DIR/myproj/.env.age) - [ "$status" -eq 0 ] - make_second_identity - STORE_PUB=$(age-keygen -y "$SECRETS_DIR/key.txt") - printf '%s\n%s\n' "$STORE_PUB" "$BOB_PUB" > "$SECRETS_DIR/recipients.txt" - # Bob cannot read the old single-key blob yet. - run age -d -i "$TEST_TMPDIR/bob.txt" "$SECRETS_DIR/myproj/.env.age" - [ "$status" -ne 0 ] - run "$SECRETS_BIN" reencrypt - [ "$status" -eq 0 ] - # Now he can. - run age -d -i "$TEST_TMPDIR/bob.txt" "$SECRETS_DIR/myproj/.env.age" - [ "$status" -eq 0 ] -} - -@test "rekey on a multi-recipient store keeps recipients and the same key" { - init_with_remote - create_project_dir myproj - run "$SECRETS_BIN" push - before=$(cat "$SECRETS_DIR/key.txt") - make_second_identity - STORE_PUB=$(age-keygen -y "$SECRETS_DIR/key.txt") - printf '%s\n%s\n' "$STORE_PUB" "$BOB_PUB" > "$SECRETS_DIR/recipients.txt" - run "$SECRETS_BIN" rekey - [ "$status" -eq 0 ] - # No new keypair was generated. - [ "$(cat "$SECRETS_DIR/key.txt")" = "$before" ] - # Both recipients can decrypt. - run age -d -i "$TEST_TMPDIR/bob.txt" "$SECRETS_DIR/myproj/.env.age" - [ "$status" -eq 0 ] -} - -@test "rekey on a legacy store still rotates to a new key (unchanged)" { - init_with_remote - create_project_dir myproj - run "$SECRETS_BIN" push - before=$(cat "$SECRETS_DIR/key.txt") - run "$SECRETS_BIN" rekey - [ "$status" -eq 0 ] - [ "$(cat "$SECRETS_DIR/key.txt")" != "$before" ] - run age -d -i "$SECRETS_DIR/key.txt" "$SECRETS_DIR/myproj/.env.age" - [ "$status" -eq 0 ] -} -``` - -- [ ] **Step 2: Run to verify they fail** - -Run: `bats test/recipients.bats -f "reencrypt|rekey on"` -Expected: FAIL ("Unknown command: reencrypt"; multi rekey generates a new key today). - -- [ ] **Step 3: Add `_reencrypt_all`** (place just before `cmd_rekey`): - -```bash -# Decrypt every blob in the store with the local key and re-encrypt each to the -# currently-loaded RECIPIENT_ARGS, then commit + push. The caller MUST have run -# _load_recipients (or set RECIPIENT_ARGS) and check_key first. Aborts with the -# store untouched on any decrypt failure (you must be a current recipient). -# Shared by recipients add/rm, reencrypt, and multi-recipient rekey. -_reencrypt_all() { - local commit_msg="$1" - local tmpdir - tmpdir=$(mktemp -d) - trap 'rm -rf "${tmpdir:-}"' EXIT INT TERM - - info "Decrypting all blobs with your key..." - local file_count=0 dir project f rel dest - for dir in "$SECRETS_DIR"/*/; do - [ -d "$dir" ] || continue - project=$(basename "$dir") - case "$project" in .*) continue ;; esac - mkdir -p "$tmpdir/$project" - while IFS= read -r f; do - [ -f "$f" ] || continue - rel=${f#"$dir"}; rel=${rel%.age} - dest="$tmpdir/$project/$rel" - mkdir -p "$(dirname "$dest")" - if ! age -d -i "$KEY_FILE" -o "$dest" "$f"; then - die "Decryption failed for $project/$rel (are you a current recipient?). Aborted; store unchanged." - fi - file_count=$((file_count + 1)) - done < <(find "$dir" -type f -name '*.age') - done - - if [ "$file_count" -eq 0 ]; then - rm -rf "$tmpdir"; trap - EXIT INT TERM - info "No encrypted blobs in the store — nothing to re-encrypt." - return 0 - fi - - local rc=$(( ${#RECIPIENT_ARGS[@]} / 2 )) - info "Re-encrypting $file_count blob(s) to $rc recipient(s)..." - for dir in "$tmpdir"/*/; do - [ -d "$dir" ] || continue - project=$(basename "$dir") - mkdir -p "$SECRETS_DIR/$project" - while IFS= read -r f; do - [ -f "$f" ] || continue - rel=${f#"$dir"} - mkdir -p "$(dirname "$SECRETS_DIR/$project/$rel")" - age "${RECIPIENT_ARGS[@]}" -o "$SECRETS_DIR/$project/${rel}.age" "$f" - done < <(find "$dir" -type f) - done - - ensure_store_protections - git -C "$SECRETS_DIR" add -A - git -C "$SECRETS_DIR" commit -m "$commit_msg" >/dev/null - if git -C "$SECRETS_DIR" remote get-url origin >/dev/null 2>&1; then - git -C "$SECRETS_DIR" push >/dev/null 2>&1 - info "Pushed re-encrypted secrets to remote" - else - info "Committed re-encrypted secrets locally (no remote configured)" - fi - rm -rf "$tmpdir"; trap - EXIT INT TERM -} - -cmd_reencrypt() { - check_cmd age - check_cmd git - resolve_store - check_initialized - check_key - _load_recipients - _reencrypt_all "reencrypt: re-encrypt all to current recipients" -} -``` - -- [ ] **Step 4: Branch `cmd_rekey`.** Replace the head of `cmd_rekey` — from its `check_cmd age` line down to and including the `info "Decrypting all files with current key..."` line — with the block below. **Leave the rest of the existing legacy body (temp dir, decrypt loop, keygen, re-encrypt loop, commit/push) exactly as-is** below this insertion: - -```bash -cmd_rekey() { - check_cmd age - check_cmd git - resolve_store - check_initialized - check_key - - # EGB-283: on a multi-recipient store, rekey means "re-encrypt every blob to - # the current recipients.txt set" — NOT a new keypair (rotating an identity is - # the member's own age-keygen + recipients rm/add). Legacy stores (no - # recipients.txt) keep the original generate-new-keypair behavior below. - if [ -e "$RECIPIENTS_FILE" ]; then - _load_recipients - info "Multi-recipient store — re-encrypting to $RECIPIENTS_FILE_NAME (no new key generated)." - _reencrypt_all "rekey: re-encrypt all to current recipients" - return 0 - fi - - # ── Legacy single-key rotation (unchanged) ── - info "Decrypting all files with current key..." -``` - -- [ ] **Step 5: Wire dispatch.** Add near `rekey)`: -```bash - reencrypt) cmd_reencrypt ;; -``` - -- [ ] **Step 6: Run the tests** - -Run: `bats test/recipients.bats -f "reencrypt|rekey"` -Expected: PASS (3 tests). - -- [ ] **Step 7: Run the full suite** (the legacy rekey tests in `secrets.bats` must still pass) - -Run: `bats test/` -Expected: PASS. - -- [ ] **Step 8: Commit** - -```bash -git add secrets test/recipients.bats -git commit -m "feat: shared _reencrypt_all + reencrypt cmd + dual rekey (EGB-283)" -``` - ---- - -### Task 4: `secrets recipients add` - -**Files:** -- Modify: `secrets` (`_recipients_add`, `_validate_recipient_name`; extend `cmd_recipients` case) -- Test: `test/recipients.bats` - -**Interfaces:** -- Produces: `_recipients_add [--name