feat: store-format-v2 self-describing migration (EGB-703, folds in EGB-700)
Stage 2 of EGB-677. Makes the store self-describing and unifies the legacy external `properties` blob suffix, via a copy-forward migration that never destroys data until an explicit, gated finalize. Scope decision (see eureka): the EGB-677 CEO plan's "flatten dotenv blobs to basename" was dropped as LOSSY — it discards the restore relpath that makes the store self-describing and adds basename collisions. Engineering analysis (4 parallel design agents) showed the store is already relpath-self-describing; the only real v1→v2 delta is the `properties` suffix. This implements the minimal, safe v2 that achieves the epic's self-describing goal. What's added: - `.secrets-format` marker (committed, one line `2`). Absence ⇒ v1 (every pre-EGB-703 store). `_store_format()` reads it; `init` stamps fresh stores born-v2. `secrets which` prints `format: vN` (EGB-700 folded in). - `_external_blob_suffix(type)` — single source of truth for the external suffix (v2: gradle-properties → properties; file unchanged). push/pull/verify all route through it, so v1 and v2 stores never disagree on blob location. - `secrets migrate` — per-project copy-forward (writes `.properties.age` twins beside v1 blobs; idempotent; needs the project manifest), `--dry-run` (reports old→new, writes nothing), `--finalize` (store-wide, the only destructive step: gates on `verify --all` green + every v1 blob twinned, cuts a `pre-v2-migrate-<sha>` recovery tag, stamps the marker, then drops v1 blobs; refuses without `--yes`/operator confirmation). rekey and verify --all stay format-agnostic (recursive find walk) — no change. 21 new bats tests (test/migrate.bats): marker/born-v2, format-aware suffix, v1 back-compat, dry-run, copy-forward idempotency, no-manifest die, finalize gates (verify-not-green refusal, untwinned refusal, recovery tag, confirmation), and full v1→window→finalize round-trip. Updated 4 existing tests for the born-v2 suffix. Full suite 231/231, bash 3.2 clean.
This commit is contained in:
parent
144ff3692b
commit
e2ad661da5
6 changed files with 487 additions and 10 deletions
|
|
@ -56,12 +56,13 @@ skips security specialist + red team, and Step 11 skips adversarial review.
|
|||
|
||||
## Architecture
|
||||
|
||||
Single bash script (`secrets`) with subcommands: init, push, pull, list, rm, rekey, verify.
|
||||
Single bash script (`secrets`) with subcommands: init, push, pull, list, rm, rekey, verify, migrate.
|
||||
|
||||
- Encryption: `age` with key files (not passphrases — age passphrases are non-scriptable)
|
||||
- Storage: Private git repo at `~/.secrets/`
|
||||
- Convention: Tracks `.env`, `.env.*`, and `.dev.vars` (not `.envrc`, `.environment-*`)
|
||||
- Manifest (EGB-677 stage 1): committed `.secrets.json` is the source of truth for what syncs — `dotenv[]` (project-relative, nested ok, `@` allowed; rail rejects `..`/absolute/symlink) + `external[]` (`properties`/`file`). Push discovery auto-adds (gated by committed `options.autoAdd`, default ON; `--frozen`/`--dry-run` overrides), bootstraps the manifest on first push (written only after ≥1 blob encrypts), and absorbs a legacy `.secrets-files` (gradle-properties → `properties`; on pull the legacy file is superseded with a warning). v1 store layout unchanged in stage 1: nested entries land at `<project>/<relpath>.age`; `properties` blobs keep the legacy `.gradle-properties.age` suffix until the stage-2 store migration. jq is a hard dep only when a manifest exists/is written; manifest-less projects run jq-free (manifest features skipped with a notice). `check_cmd` prints platform-aware install hints.
|
||||
- Manifest (EGB-677 stage 1): committed `.secrets.json` is the source of truth for what syncs — `dotenv[]` (project-relative, nested ok, `@` allowed; rail rejects `..`/absolute/symlink) + `external[]` (`properties`/`file`). Push discovery auto-adds (gated by committed `options.autoAdd`, default ON; `--frozen`/`--dry-run` overrides), bootstraps the manifest on first push (written only after ≥1 blob encrypts), and absorbs a legacy `.secrets-files` (gradle-properties → `properties`; on pull the legacy file is superseded with a warning). Store layout: nested dotenv entries land at `<project>/<relpath>.age` (relpath preserved — the store self-describes where a file restores). jq is a hard dep only when a manifest exists/is written; manifest-less projects run jq-free (manifest features skipped with a notice). `check_cmd` prints platform-aware install hints.
|
||||
- Store format (EGB-677 stage 2 / EGB-703): the store is self-describing via a committed one-line `$SECRETS_DIR/.secrets-format` file (`2`). Absence ⇒ v1 (every store predating EGB-703). v2's only on-disk change vs v1 is the external `properties` blob suffix: `.gradle-properties.age` → `.properties.age` (matching the manifest `type`); dotenv and `file` blobs are unchanged. `_store_format()` reads the marker; `_external_blob_suffix(type)` is the single source of truth for the suffix (push/pull/verify all route through it, so v1 and v2 stores never disagree on where a blob lives). `init` stamps a fresh store v2 (born-v2). `secrets which` prints `format: vN`. **Migration is copy-forward and non-destructive:** `secrets migrate --dry-run` (per project, reports old→new, writes nothing) → `secrets migrate` (per project, writes `.properties.age` twins beside the v1 blobs; needs the project manifest to know which externals are `properties`; idempotent) → `secrets migrate --finalize` (store-wide; the ONLY destructive step — gates on `verify --all` green + every v1 blob having a v2 twin, cuts a `pre-v2-migrate-<sha>` recovery tag, stamps the marker, then drops v1 blobs; refuses without `--yes`/operator confirmation since a lagging v1 client against a finalized store stops seeing `properties` externals until it upgrades). The deliberate flatten-to-basename naming the EGB-677 CEO plan sketched was dropped as lossy (it discards the restore relpath that makes the store self-describing) — see the EGB-703 eureka.
|
||||
- Verify (EGB-698): `secrets verify` is a read-only integrity check. Default mode (current project) cross-checks `$PWD/.secrets.json` against `$SECRETS_DIR/<project>/` both ways (declared-but-missing blobs + orphaned blobs) and decrypt-tests every blob (dotenv + external) by streaming plaintext to `/dev/null` (never written to disk). `secrets verify --all` decrypt-tests every blob in every project (integrity only — the store carries no manifests, so consistency can't be checked store-wide). Both recurse the whole project tree (`find -type f`, same as rekey/list). Exits non-zero on any finding so it can gate the stage-2 `migrate --finalize` and CI. The store deliberately holds no manifest — `.secrets.json` is committed in each project's own repo and read from `$PWD`.
|
||||
- External files: `.secrets-files` manifest tracks designated keys from files outside the project (e.g. `~/.gradle/gradle.properties`, merged not overwritten — EGB-531) and whole binary files (type `file`, e.g. an Android upload keystore — EGB-652); see below
|
||||
- Workspaces: `--workspaces` flag reads `package.json` workspaces, requires `jq`
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue