Merge pull request 'v0.7.0.0 feat: additive-v2 dual-write — optional finalize (EGB-712)' (#8) from brian/egb-712-secrets-additive-v2-dual-write-defuse-the-finalize into main

This commit is contained in:
EGBT Technologies 2026-06-08 20:01:47 +02:00
commit 40635f9e1f
8 changed files with 993 additions and 31 deletions

View file

@ -5,6 +5,29 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to a four-digit MAJOR.MINOR.PATCH.MICRO version scheme. and this project adheres to a four-digit MAJOR.MINOR.PATCH.MICRO version scheme.
## [0.7.0.0] - 2026-06-08
### Changed
- **Additive store-format v2 (EGB-712)** — upgraded `secrets` clients now read
either external blob suffix (`.properties.age` or the legacy
`.gradle-properties.age`) and **dual-write** a `properties` external whenever a
v1 twin already exists in the store. Existing externals keep working for
teammates on an older `secrets`; only a brand-new `properties` external is
written v2-only (a gentle "upgrade to see it" forcing function). dotenv and
whole-`file` externals are unchanged across formats and always propagate.
- **`secrets migrate --finalize` is now optional GC**, not a required milestone.
Because clients dual-write and read-fall-back, no teammate is ever cut off by
*not* finalizing; finalize only reclaims the duplicate v1 blobs, and stays
deferrable indefinitely. Its safety gates are unchanged. This defuses the
cross-machine "all clients must be v2 before finalize" coordination gate.
### Added
- **`secrets migrate --status`** now reports `v2-only` externals per project
(the ones an un-upgraded client cannot read), so you can see the forcing
function's footprint at a glance.
## [0.6.1.0] - 2026-06-08 ## [0.6.1.0] - 2026-06-08
### Changed ### Changed

View file

@ -62,7 +62,7 @@ Single bash script (`secrets`) with subcommands: init, push, pull, list, rm, rek
- Storage: Private git repo at `~/.secrets/` - Storage: Private git repo at `~/.secrets/`
- Convention: Tracks `.env`, `.env.*`, and `.dev.vars` (not `.envrc`, `.environment-*`) - Convention: Tracks `.env`, `.env.*`, and `.dev.vars` (not `.envrc`, `.environment-*`)
- Manifest (EGB-677 stage 1): committed `.secrets.json` is the source of truth for what syncs — `dotenv[]` (project-relative, nested ok, `@` allowed; rail rejects `..`/absolute/symlink) + `external[]` (`properties`/`file`). Push discovery auto-adds (gated by committed `options.autoAdd`, default ON; `--frozen`/`--dry-run` overrides), bootstraps the manifest on first push (written only after ≥1 blob encrypts), and absorbs a legacy `.secrets-files` (gradle-properties → `properties`; on pull the legacy file is superseded with a warning). Store layout: nested dotenv entries land at `<project>/<relpath>.age` (relpath preserved — the store self-describes where a file restores). jq is a hard dep only when a manifest exists/is written; manifest-less projects run jq-free (manifest features skipped with a notice). `check_cmd` prints platform-aware install hints. - Manifest (EGB-677 stage 1): committed `.secrets.json` is the source of truth for what syncs — `dotenv[]` (project-relative, nested ok, `@` allowed; rail rejects `..`/absolute/symlink) + `external[]` (`properties`/`file`). Push discovery auto-adds (gated by committed `options.autoAdd`, default ON; `--frozen`/`--dry-run` overrides), bootstraps the manifest on first push (written only after ≥1 blob encrypts), and absorbs a legacy `.secrets-files` (gradle-properties → `properties`; on pull the legacy file is superseded with a warning). Store layout: nested dotenv entries land at `<project>/<relpath>.age` (relpath preserved — the store self-describes where a file restores). jq is a hard dep only when a manifest exists/is written; manifest-less projects run jq-free (manifest features skipped with a notice). `check_cmd` prints platform-aware install hints.
- Store format (EGB-677 stage 2 / EGB-703): the store is self-describing via a committed one-line `$SECRETS_DIR/.secrets-format` file (`2`). Absence ⇒ v1 (every store predating EGB-703). v2's only on-disk change vs v1 is the external `properties` blob suffix: `.gradle-properties.age``.properties.age` (matching the manifest `type`); dotenv and `file` blobs are unchanged. `_store_format()` reads the marker; `_external_blob_suffix(type)` is the single source of truth for the suffix (push/pull/verify all route through it, so v1 and v2 stores never disagree on where a blob lives). `init` stamps a fresh store v2 (born-v2). `secrets which` prints the store-format line `format: vN`, and (EGB-700) when a `.secrets.json` is present the manifest header line also carries its schema version (`manifest (.secrets.json at <path>, version N):`). **Migration is copy-forward and non-destructive:** `secrets migrate --dry-run` (per project, reports old→new, writes nothing) → `secrets migrate` (per project, manifest-free: enumerates the store's `*.gradle-properties.age` blobs directly — same source of truth as `--finalize` — and writes their `.properties.age` twins, so a legacy `.secrets-files`-only project with no `.secrets.json` migrates cleanly and no store blob is left un-twinned; idempotent; EGB-710) → `secrets migrate --finalize` (store-wide; the ONLY destructive step — gates on `verify --all` green + every v1 blob having a v2 twin, cuts a `pre-v2-migrate-<sha>` recovery tag, stamps the marker, then drops v1 blobs; refuses without `--yes`/operator confirmation since a lagging v1 client against a finalized store stops seeing `properties` externals until it upgrades). `secrets migrate --status` is a read-only survey that walks every project in the store and reports each one's v2 readiness (v2-ready / migrated / NEEDS MIGRATE), exiting non-zero while any v1 blob is un-twinned so it gates the path to `--finalize` (EGB-710). The deliberate flatten-to-basename naming the EGB-677 CEO plan sketched was dropped as lossy (it discards the restore relpath that makes the store self-describing) — see the EGB-703 eureka. - Store format (EGB-677 stage 2 / EGB-703): the store is self-describing via a committed one-line `$SECRETS_DIR/.secrets-format` file (`2`). Absence ⇒ v1 (every store predating EGB-703). v2's only on-disk change vs v1 is the external `properties` blob suffix: `.gradle-properties.age``.properties.age` (matching the manifest `type`); dotenv and `file` blobs are unchanged. `_store_format()` reads the marker. **Additive v2 (EGB-712):** reads resolve a `properties` blob by trying `.properties.age` then falling back to `.gradle-properties.age` (`_resolve_external_blob_read`); writes dual-write a `properties` external only when a v1 twin already exists in the store (`_external_blob_write_targets`), so existing externals keep old clients fresh while brand-new externals are written v2-only (a gentle forcing function). Blob location no longer depends on the marker — the old `_external_blob_suffix` is gone. `init` stamps a fresh store v2 (born-v2). `secrets which` prints the store-format line `format: vN`, and (EGB-700) when a `.secrets.json` is present the manifest header line also carries its schema version (`manifest (.secrets.json at <path>, version N):`). **Migration is copy-forward and non-destructive:** `secrets migrate --dry-run` (per project, reports old→new, writes nothing) → `secrets migrate` (per project, manifest-free: enumerates the store's `*.gradle-properties.age` blobs directly — same source of truth as `--finalize` — and writes their `.properties.age` twins, so a legacy `.secrets-files`-only project with no `.secrets.json` migrates cleanly and no store blob is left un-twinned; idempotent; EGB-710) → `secrets migrate --finalize` (store-wide; the ONLY destructive step — gates on `verify --all` green + every v1 blob having a v2 twin, cuts a `pre-v2-migrate-<sha>` recovery tag, stamps the marker, then drops v1 blobs; refuses without `--yes`/operator confirmation since a lagging v1 client against a finalized store stops seeing `properties` externals until it upgrades). `secrets migrate --status` is a read-only survey that walks every project in the store and reports each one's v2 readiness (v2-ready / migrated / NEEDS MIGRATE, plus a `v2-only` count of externals old clients can't read), exiting non-zero while any v1 blob is un-twinned so it gates the path to `--finalize` (EGB-710/EGB-712). **Under additive v2 (EGB-712) `--finalize` is now OPTIONAL GC, not a required milestone:** because upgraded clients dual-write existing externals and read-fall-back, *not* finalizing never cuts anyone off — finalize only reclaims the duplicate v1 blobs and stays deferrable indefinitely (defusing the cross-machine coordination gate). dotenv and `file` blobs are identical across formats, so they always propagate to old clients; only a brand-new `properties` external is v2-only. The deliberate flatten-to-basename naming the EGB-677 CEO plan sketched was dropped as lossy (it discards the restore relpath that makes the store self-describing) — see the EGB-703 eureka.
- Verify (EGB-698): `secrets verify` is a read-only integrity check. Default mode (current project) cross-checks `$PWD/.secrets.json` against `$SECRETS_DIR/<project>/` both ways (declared-but-missing blobs + orphaned blobs) and decrypt-tests every blob (dotenv + external) by streaming plaintext to `/dev/null` (never written to disk). `secrets verify --all` decrypt-tests every blob in every project (integrity only — the store carries no manifests, so consistency can't be checked store-wide). Both recurse the whole project tree (`find -type f`, same as rekey/list). Exits non-zero on any finding so it can gate the stage-2 `migrate --finalize` and CI. The store deliberately holds no manifest — `.secrets.json` is committed in each project's own repo and read from `$PWD`. - Verify (EGB-698): `secrets verify` is a read-only integrity check. Default mode (current project) cross-checks `$PWD/.secrets.json` against `$SECRETS_DIR/<project>/` both ways (declared-but-missing blobs + orphaned blobs) and decrypt-tests every blob (dotenv + external) by streaming plaintext to `/dev/null` (never written to disk). `secrets verify --all` decrypt-tests every blob in every project (integrity only — the store carries no manifests, so consistency can't be checked store-wide). Both recurse the whole project tree (`find -type f`, same as rekey/list). Exits non-zero on any finding so it can gate the stage-2 `migrate --finalize` and CI. The store deliberately holds no manifest — `.secrets.json` is committed in each project's own repo and read from `$PWD`.
- External files: `.secrets-files` manifest tracks designated keys from files outside the project (e.g. `~/.gradle/gradle.properties`, merged not overwritten — EGB-531) and whole binary files (type `file`, e.g. an Android upload keystore — EGB-652); see below - External files: `.secrets-files` manifest tracks designated keys from files outside the project (e.g. `~/.gradle/gradle.properties`, merged not overwritten — EGB-531) and whole binary files (type `file`, e.g. an Android upload keystore — EGB-652); see below
- Workspaces: `--workspaces` flag reads `package.json` workspaces, requires `jq` - Workspaces: `--workspaces` flag reads `package.json` workspaces, requires `jq`

View file

@ -173,7 +173,20 @@ secrets clear
| `secrets verify --all` | Decrypt-test every blob in every project — a store-wide integrity sweep | | `secrets verify --all` | Decrypt-test every blob in every project — a store-wide integrity sweep |
| `secrets migrate [--dry-run]` | Copy-forward this project's encrypted blobs to store format v2 (non-destructive; manifest-free; `--dry-run` previews) | | `secrets migrate [--dry-run]` | Copy-forward this project's encrypted blobs to store format v2 (non-destructive; manifest-free; `--dry-run` previews) |
| `secrets migrate --status` | Survey every project's v2 readiness; exits non-zero until the whole store is finalize-ready | | `secrets migrate --status` | Survey every project's v2 readiness; exits non-zero until the whole store is finalize-ready |
| `secrets migrate --finalize` | Drop the old v1 blobs and mark the store v2 — runs once, store-wide, after `verify` is green and every machine is upgraded | | `secrets migrate --finalize` | **Optional GC** — drop the old v1 blobs and mark the store pure v2. Never required: upgraded clients dual-write and read-fall-back, so not finalizing never cuts anyone off |
### Upgrading: do teammates on an older `secrets` get new secrets?
Store-format v2 is **additive** — an upgraded client reads either blob suffix and keeps the old (v1) suffix alive for externals that already existed, so you almost never have to coordinate an upgrade:
| Secret type | Old client gets it? |
|---|---|
| `.env` / `.env.*` / `.dev.vars` | **Yes, always** (blob name is identical across formats) |
| whole-file external (`file`) | **Yes, always** |
| `properties` external that already existed | **Yes** (dual-written so old clients stay fresh) |
| brand-new `properties` external | **No — must upgrade `secrets`** (the gentle forcing function) |
"Upgrade your secrets" = `git pull` the tool clone (binary ≥ 0.6.0.0) and/or `secrets migrate` the store. A read-only teammate only needs the tool `git pull`.
### Automatic project detection ### Automatic project detection

View file

@ -1 +1 @@
0.6.1.0 0.7.0.0

View file

@ -0,0 +1,587 @@
# Additive-v2 Dual-Write Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Make store-format v2 *additive* — upgraded `secrets` clients read either blob suffix and dual-write existing `properties` externals — so the destructive `migrate --finalize` becomes optional GC and the cross-machine coordination gate disappears.
**Architecture:** Replace the marker-driven single-suffix helper `_external_blob_suffix(type)` with two helpers: a read-resolver that tries `.properties.age` then falls back to `.gradle-properties.age`, and a write-targets helper that writes the v2 suffix always plus the v1 suffix *only when a v1 twin already exists* (dual-write existing externals; brand-new externals are v2-only — the intended forcing function). Reads and writes no longer depend on the `.secrets-format` marker, which keeps its meaning (born-v2 / finalized). Because fresh pushes now write the v2 suffix on any store, the migrate/finalize test fixtures (which relied on push producing a v1 blob) are updated to fabricate an old-client v1 blob.
**Tech Stack:** Single bash 3.2 script (`secrets`); `age`, `git`, `jq`. Tests: `bats-core` (`test/migrate.bats`, `test/manifest.bats`). Every standalone `[[ ]]` test assertion ends with `|| false` (bash 3.2 ERR-trap gotcha).
**Spec:** `docs/superpowers/specs/2026-06-08-additive-v2-dual-write-design.md`. **Ticket:** EGB-712.
---
## Background facts (verified against branch `brian/egb-712-...`, post-EGB-710)
- `_external_blob_suffix()``secrets:548-555`. Returns `properties` for `gradle-properties` on a v2 store (`_store_format == 2`), else the type verbatim; `file` always returns `file`. Blob path = `external/<slug>.<suffix>.age`.
- Callers of `_external_blob_suffix`: push file write `secrets:655`, push properties write `secrets:684`, pull read `secrets:713`, verify read `secrets:2056`. All four are replaced; then the helper is deleted.
- `_secrets_files_slug(path)` (`secrets:512`) derives the machine-independent slug. dotenv and `file` blobs are byte-identical in v1/v2 (only the `properties` suffix diverges).
- `_store_format()` (`secrets:529`) and the marker stay as-is — set only by `init` (born-v2) and `migrate --finalize`. Push must NOT stamp it (see spec §3).
- Existing test that codifies OLD write behavior: `test/migrate.bats:52` "push on a v1 store still writes .gradle-properties.age (back-compat)" — rewritten in Task 2.
- Tests that fabricate-or-rely-on a v1 properties blob from `make_v1_store; push` and break once push writes v2-only (repaired in Task 3): the migrate copy-forward/idempotent tests, the `--dry-run` rename test, the dotenv+file untouched test, all four finalize tests, the EGB-710 manifest-free / undeclared-twin tests, and the two `--status` tests that need an actual v1 blob. (dotenv-only and already-v2 tests are unaffected.)
- Baseline before this plan: `bats test/` = 242 passing.
## File structure
- Modify: `secrets` — delete `_external_blob_suffix` (548-555); add `_resolve_external_blob_read` + `_external_blob_write_targets` in its place; rewire push (651-688), pull (712-713), verify (2052-2068); extend `_migrate_status`; docs in `cmd_help`.
- Modify: `test/migrate.bats` — add `m_fake_v1_blob` helper; add read-fallback + write-rule + status-coverage tests; repair the v1-blob-dependent fixtures.
- Modify: `CLAUDE.md`, `README.md`, `VERSION` (→ `0.7.0.0`), `CHANGELOG.md`.
---
## Task 1: Read-resolver — reads try both suffixes
**Files:** `secrets` (replace `_external_blob_suffix` with the read-resolver; rewire pull + verify reads), `test/migrate.bats`.
- [ ] **Step 1: Write the failing test** — a v2 store whose properties blob exists ONLY in the v1 suffix must still pull.
Add to `test/migrate.bats` (after the format-marker tests, ~line 60):
```bash
@test "pull reads a v1-suffix properties blob on a v2 store (read-fallback)" {
init_with_remote # born-v2 store (marker=2)
m_gradle_src $'beaconClerkPkTest=pk_test_v1\n'
create_project_dir rffallback
printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files
"$SECRETS_BIN" push rffallback >/dev/null 2>&1 # writes .properties.age on a v2 store
# Simulate an external that exists only in the v1 suffix (an old client wrote it):
local v2blob; v2blob=$(ls "$SECRETS_DIR/rffallback/external/"*.properties.age)
mv "$v2blob" "${v2blob%.properties.age}.gradle-properties.age"
rm -f "$HOME/.gradle/gradle.properties"
"$SECRETS_BIN" pull rffallback >/dev/null 2>&1
run grep -q 'beaconClerkPkTest=pk_test_v1' "$HOME/.gradle/gradle.properties"
[ "$status" -eq 0 ]
}
```
- [ ] **Step 2: Run it, confirm it fails**
Run: `bats test/migrate.bats -f "read-fallback"`
Expected: FAIL — old pull (`secrets:713`) uses `_external_blob_suffix gradle-properties` = `properties` on a v2 store, looks only for `.properties.age` (which we renamed away), warns "no encrypted data", restores nothing → the grep fails.
- [ ] **Step 3: Replace `_external_blob_suffix` with the read-resolver.** Replace `secrets:542-555` (the comment block + `_external_blob_suffix()` through its closing `}`) with:
```bash
# Resolve the on-disk path of an external blob for READING. Tries the v2 suffix
# (.properties.age) first, then falls back to the v1 (.gradle-properties.age) for
# `properties` externals, so an upgraded client finds the blob whichever format
# wrote it (additive v2 — EGB-712). `file` externals share one suffix in both
# formats. Echoes the path of the blob that exists; if neither exists, echoes the
# canonical v2 path so the caller's "no blob" message reads sensibly. Read-only.
_resolve_external_blob_read() {
local project="$1" slug="$2" mtype="$3"
local base="$SECRETS_DIR/$project/external/$slug"
case "$mtype" in
file)
echo "$base.file.age" ;;
properties|gradle-properties)
if [ -f "$base.properties.age" ]; then
echo "$base.properties.age"
elif [ -f "$base.gradle-properties.age" ]; then
echo "$base.gradle-properties.age"
else
echo "$base.properties.age"
fi ;;
*)
echo "$base.$mtype.age" ;;
esac
}
```
(The write-targets helper is added in Task 2 — leave a gap; do not reintroduce `_external_blob_suffix`.)
- [ ] **Step 4: Rewire the pull read.** At `secrets:712-713`, replace:
```bash
local slug; slug=$(_secrets_files_slug "$mpath")
local blob="$SECRETS_DIR/$project/external/$slug.$(_external_blob_suffix "$mtype").age"
```
with:
```bash
local slug; slug=$(_secrets_files_slug "$mpath")
local blob; blob=$(_resolve_external_blob_read "$project" "$slug" "$mtype")
```
- [ ] **Step 5: Rewire the verify read.** At `secrets:2055-2065`, replace:
```bash
slug=$(_secrets_files_slug "$epath")
erel="external/$slug.$(_external_blob_suffix "$etype").age"
# Account for BOTH the v1 and v2 suffix forms in the orphan set. During the
# migration window (after copy-forward, before --finalize) the v2 twin
# coexists with the v1 blob; neither should read as an orphan whichever
# format the store currently reports. (file's two forms are identical.)
expected="${expected}external/$slug.$etype.age"$'\n'
[ "$etype" = "gradle-properties" ] && expected="${expected}external/$slug.properties.age"$'\n'
eblob="$pdir/$erel"
if [ ! -f "$eblob" ]; then
echo "FINDING: external '$epath' ($etype) is declared but has no blob in the store ($project/$erel missing). Run 'secrets push'." >&2
```
with:
```bash
slug=$(_secrets_files_slug "$epath")
# Account for BOTH suffix forms in the orphan set — a dual-written `properties`
# external (additive v2 — EGB-712) legitimately has both blobs on disk; neither
# is an orphan. (file's two forms are identical.)
expected="${expected}external/$slug.$etype.age"$'\n'
[ "$etype" = "gradle-properties" ] && expected="${expected}external/$slug.properties.age"$'\n'
eblob=$(_resolve_external_blob_read "$project" "$slug" "$etype")
erel="${eblob#"$pdir"/}"
if [ ! -f "$eblob" ]; then
echo "FINDING: external '$epath' ($etype) is declared but has no blob in the store ($project/$erel missing). Run 'secrets push'." >&2
```
(Note: `_external_blob_suffix` still has two remaining callers in push — push isn't rewired until Task 2, so the script still parses and runs. Those calls keep working because the function is only deleted in Task 2 Step 6, after push is rewired.)
**IMPORTANT:** do NOT delete `_external_blob_suffix` yet — push (`secrets:655`, `secrets:684`) still calls it until Task 2. Deleting it now breaks push.
- [ ] **Step 6: Run the read-fallback test + full migrate suite**
Run: `bats test/migrate.bats`
Expected: the new "read-fallback" test PASSES; all other migrate tests still PASS (push unchanged; pull/verify now use the resolver, which is equivalent to the old behavior whenever the suffix matches the store format).
- [ ] **Step 7: Commit**
```bash
git add secrets test/migrate.bats
git commit -m "feat: read-resolver tries both external suffixes (additive v2, EGB-712)"
```
---
## Task 2: Write-targets — twin rule (dual-write existing, v2-only for new)
**Files:** `secrets` (add `_external_blob_write_targets`; rewire push file + properties writes; delete `_external_blob_suffix`), `test/migrate.bats`.
- [ ] **Step 1: Write the failing tests.** Add to `test/migrate.bats`:
```bash
@test "push writes the v2 suffix for a fresh external even on a v1 store" {
make_v1_store
m_gradle_src $'beaconClerkPkTest=pk_test_abc\n'
create_project_dir freshv1
printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files
"$SECRETS_BIN" push freshv1 >/dev/null 2>&1
run bash -c "ls $SECRETS_DIR/freshv1/external/*.properties.age"
[ "$status" -eq 0 ] # v2 suffix regardless of marker
run bash -c "ls $SECRETS_DIR/freshv1/external/*.gradle-properties.age 2>/dev/null"
[ "$status" -ne 0 ] # no v1 twin for a brand-new external
}
@test "push dual-writes the v1 twin so old clients stay fresh" {
make_v1_store
m_gradle_src $'beaconClerkPkTest=pk_test_old\n'
create_project_dir dualwrite
printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files
"$SECRETS_BIN" push dualwrite >/dev/null 2>&1 # v2-only (fresh)
m_fake_v1_twin dualwrite # simulate a pre-existing v1 twin
m_gradle_src $'beaconClerkPkTest=pk_test_new\n'
"$SECRETS_BIN" push dualwrite >/dev/null 2>&1 # twin exists -> dual-write both
# Prove the v1 twin was refreshed: drop the v2 blob, pull, expect the NEW value.
rm -f "$SECRETS_DIR/dualwrite/external/"*.properties.age
rm -f "$HOME/.gradle/gradle.properties"
"$SECRETS_BIN" pull dualwrite >/dev/null 2>&1
run grep -q 'beaconClerkPkTest=pk_test_new' "$HOME/.gradle/gradle.properties"
[ "$status" -eq 0 ]
}
```
- [ ] **Step 2: Add the `m_fake_v1_twin` test helper.** In `test/migrate.bats`, next to `make_v1_store` (~line 12), add:
```bash
# Simulate an old (v1) client's properties blob: copy the pushed v2
# .properties.age to its v1 .gradle-properties.age twin. (Current clients never
# write the v1 suffix for a fresh external, so tests fabricate it.) Use `cp` to
# KEEP the v2 blob (dual present); see m_make_v1_only to leave only the v1 blob.
m_fake_v1_twin() {
local proj="$1" v2
v2=$(ls "$SECRETS_DIR/$proj/external/"*.properties.age)
cp "$v2" "${v2%.properties.age}.gradle-properties.age"
}
# Like m_fake_v1_twin but renames (leaves ONLY the v1 blob) — for old-client-only
# / copy-forward fixtures.
m_make_v1_only() {
local proj="$1" v2
v2=$(ls "$SECRETS_DIR/$proj/external/"*.properties.age)
mv "$v2" "${v2%.properties.age}.gradle-properties.age"
}
```
- [ ] **Step 3: Run the new tests, confirm they fail**
Run: `bats test/migrate.bats -f "fresh external even on a v1 store"`
Expected: FAIL — on a v1 store, the old push (`_external_blob_suffix gradle-properties` = `gradle-properties`) writes `.gradle-properties.age`, so the `*.properties.age` assertion fails.
Run: `bats test/migrate.bats -f "dual-writes the v1 twin"`
Expected: FAIL — old push writes a single suffix; the second push won't refresh the v1 twin.
- [ ] **Step 4: Add the write-targets helper.** Immediately after `_resolve_external_blob_read` (added in Task 1), insert:
```bash
# The on-disk path(s) to WRITE for an external blob, one per line. For a
# `properties` external this is the v2 suffix (.properties.age) ALWAYS, plus the
# v1 suffix (.gradle-properties.age) WHEN a v1 twin already exists in the store
# (dual-write keeps old clients fresh; a brand-new external is v2-only — the
# intended forcing function, additive v2 / EGB-712). `file` externals have a
# single suffix in both formats. Independent of the store marker.
_external_blob_write_targets() {
local project="$1" slug="$2" mtype="$3"
local base="$SECRETS_DIR/$project/external/$slug"
case "$mtype" in
file)
echo "$base.file.age" ;;
properties|gradle-properties)
echo "$base.properties.age"
[ -f "$base.gradle-properties.age" ] && echo "$base.gradle-properties.age" ;;
*)
echo "$base.$mtype.age" ;;
esac
}
```
- [ ] **Step 5: Rewire the push writes.** At `secrets:651-658` (the `file` branch), replace:
```bash
if [ "$mtype" = "file" ]; then
# EGB-652: whole-file sync — encrypt the file verbatim (binary-safe).
mkdir -p "$SECRETS_DIR/$project/external"
local fslug; fslug=$(_secrets_files_slug "$mpath")
age -r "$pubkey" -o "$SECRETS_DIR/$project/external/$fslug.$(_external_blob_suffix file).age" "$expanded"
info "Encrypted file $mpath"
pushed=$((pushed + 1))
continue
fi
```
with:
```bash
if [ "$mtype" = "file" ]; then
# EGB-652: whole-file sync — encrypt the file verbatim (binary-safe).
mkdir -p "$SECRETS_DIR/$project/external"
local fslug; fslug=$(_secrets_files_slug "$mpath")
local wt
while IFS= read -r wt; do
[ -n "$wt" ] || continue
age -r "$pubkey" -o "$wt" "$expanded"
done < <(_external_blob_write_targets "$project" "$fslug" file)
info "Encrypted file $mpath"
pushed=$((pushed + 1))
continue
fi
```
Then at `secrets:682-684` (the properties branch), replace:
```bash
mkdir -p "$SECRETS_DIR/$project/external"
local slug; slug=$(_secrets_files_slug "$mpath")
age -r "$pubkey" -o "$SECRETS_DIR/$project/external/$slug.$(_external_blob_suffix "$mtype").age" "$tmp"
```
with:
```bash
mkdir -p "$SECRETS_DIR/$project/external"
local slug; slug=$(_secrets_files_slug "$mpath")
local wt
while IFS= read -r wt; do
[ -n "$wt" ] || continue
age -r "$pubkey" -o "$wt" "$tmp"
done < <(_external_blob_write_targets "$project" "$slug" "$mtype")
```
- [ ] **Step 6: Delete the now-unused `_external_blob_suffix`.** Confirm zero remaining callers first:
Run: `grep -n "_external_blob_suffix" secrets`
Expected: no matches (all four call sites rewired). If any remain, rewire them before deleting. Then delete the `_resolve_external_blob_read`-replaced... — it's already gone (replaced in Task 1). Verify the function is absent: `grep -c "_external_blob_suffix()" secrets``0`.
- [ ] **Step 7: Run the two new write tests**
Run: `bats test/migrate.bats -f "fresh external even on a v1 store"` then `-f "dual-writes the v1 twin"`
Expected: both PASS.
- [ ] **Step 8: Rewrite the obsolete back-compat test.** Replace the `@test "push on a v1 store still writes .gradle-properties.age (back-compat)"` block (`test/migrate.bats:52-60`) with:
```bash
@test "push on a v1 store writes the v2 suffix for a fresh external (additive v2)" {
make_v1_store
m_gradle_src $'beaconClerkPkTest=pk_test_abc\n'
create_project_dir v1push
printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files
"$SECRETS_BIN" push v1push >/dev/null 2>&1
run bash -c "ls $SECRETS_DIR/v1push/external/*.properties.age"
[ "$status" -eq 0 ]
}
```
- [ ] **Step 9: Run the full migrate suite — expect the v1-blob-dependent fixtures to FAIL.** This is expected; Task 3 repairs them.
Run: `bats test/migrate.bats`
Expected: the read-fallback + two write tests + rewritten back-compat test PASS; several copy-forward/finalize/status tests now FAIL (push no longer writes a `.gradle-properties.age` for them to migrate). Note which fail — Task 3 fixes exactly those.
- [ ] **Step 10: Commit** (suite intentionally not yet fully green — Task 3 follows immediately)
```bash
git add secrets test/migrate.bats
git commit -m "feat: twin-rule write targets — dual-write existing, v2-only for new (additive v2, EGB-712)"
```
---
## Task 3: Repair migrate/finalize/status fixtures
**Files:** `test/migrate.bats`. No production code changes — this re-greens the suite by fabricating the old-client v1 blobs that push no longer writes.
The rule for each repair: after the `"$SECRETS_BIN" push <proj>` line, insert a fabrication call:
- Use **`m_make_v1_only <proj>`** (rename → only the v1 blob exists) for tests asserting a `*.gradle-properties.age` blob exists / is copy-forwarded (mirrors the pre-EGB-712 state where push produced a v1 blob).
- Use **`m_fake_v1_twin <proj>`** (keep both) only where a test needs both suffixes present.
- [ ] **Step 1: Repair the copy-forward / dry-run / idempotent tests.** In each of these tests, insert `m_make_v1_only <proj>` immediately after the `push <proj>` line:
- `"migrate --dry-run reports the rename and writes nothing"` (proj `dryproj`)
- `"migrate copy-forward creates the v2 twin and keeps the v1 blob (byte-identical)"` (proj `cfproj`)
- `"migrate copy-forward is idempotent"` (proj `idemproj`)
- `"migrate leaves dotenv and file blobs untouched"` (proj `mixproj`) — note this one ALSO pushes a `file` external; `m_make_v1_only` only touches `*.properties.age`, leaving the `.file.age` blob alone (correct).
Worked example — the copy-forward test becomes:
```bash
@test "migrate copy-forward creates the v2 twin and keeps the v1 blob (byte-identical)" {
make_v1_store
m_gradle_src $'beaconClerkPkTest=pk_test_abc\n'
create_project_dir cfproj
printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files
"$SECRETS_BIN" push cfproj >/dev/null 2>&1
m_make_v1_only cfproj
local old; old=$(ls "$SECRETS_DIR/cfproj/external/"*.gradle-properties.age)
run "$SECRETS_BIN" migrate
[ "$status" -eq 0 ]
local new; new=$(ls "$SECRETS_DIR/cfproj/external/"*.properties.age)
[ -f "$old" ]
[ -f "$new" ]
cmp -s "$old" "$new"
}
```
- [ ] **Step 2: Repair the finalize tests.** Insert `m_make_v1_only <proj>` after the `push <proj>` line in:
- `"finalize refuses when verify --all is not green"` (proj `failverify`) — then the existing `migrate` step creates the twin; corrupting `*.properties.age` still trips verify.
- `"finalize refuses an un-twinned v1 blob (project not migrated)"` (proj `untwinned`) — leaves a lone v1 blob, no twin: exactly the un-twinned state the test wants.
- `"finalize green path drops v1, keeps v2, stamps the marker"` (proj `finproj`).
- `"finalize cuts a recovery tag before deleting v1 blobs"` (proj — read it from the test).
- [ ] **Step 3: Repair the EGB-710 manifest-free / undeclared-twin tests.** Insert `m_make_v1_only <proj>` after the `push <proj>` line in:
- `"migrate copy-forwards a v1 properties blob with no .secrets.json (manifest-free)"` (proj `nomanifestblob`) — insert BEFORE the `rm -f .secrets.json` line.
- `"migrate twins a store blob even when the manifest no longer declares it"` (proj `staleblob`) — insert before the `.secrets.json` rewrite.
- [ ] **Step 4: Repair the `--status` tests that need a real v1 blob.**
- `"migrate --status flags a project that needs migrating"` (proj `needsmig`) — insert `m_make_v1_only needsmig` after push, so a lone un-twinned v1 blob exists → NEEDS MIGRATE.
- `"migrate --status reports finalize-ready once every blob is twinned"` (proj `readymig`) — insert `m_make_v1_only readymig` after push and BEFORE the `migrate` step (migrate then creates the twin → finalize-ready).
- [ ] **Step 5: Run the full migrate suite**
Run: `bats test/migrate.bats`
Expected: ALL pass. If any copy-forward test still reports "nothing to migrate", its `m_make_v1_only` call is missing or misplaced (must come after push, before migrate).
- [ ] **Step 6: Run the WHOLE suite** (manifest.bats exercises externals end-to-end and must still be green)
Run: `bats test/`
Expected: all pass. If a `manifest.bats` external test fails, check it isn't asserting a specific suffix that additive-v2 changed (a fresh push now writes `.properties.age`); update such an assertion the same way (assert `.properties.age`, or use the resolver-agnostic round-trip via pull).
- [ ] **Step 7: Commit**
```bash
git add test/migrate.bats
git commit -m "test: fabricate old-client v1 blobs in migrate/finalize/status fixtures (additive v2, EGB-712)"
```
---
## Task 4: `migrate --status` — dual-write coverage line
**Files:** `secrets` (`_migrate_status`, `secrets:2190-2227`), `test/migrate.bats`.
Adds visibility into which `properties` externals are v2-only (old clients can't read them — the forcing function) vs dual-written (old clients still served).
- [ ] **Step 1: Write the failing test.** Add to `test/migrate.bats`:
```bash
@test "migrate --status counts v2-only externals (old clients not served)" {
make_v1_store
m_gradle_src $'beaconClerkPkTest=pk_test_abc\n'
create_project_dir v2onlyext
printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files
"$SECRETS_BIN" push v2onlyext >/dev/null 2>&1 # v2-only (fresh, no v1 twin)
run "$SECRETS_BIN" migrate --status
[ "$status" -eq 0 ] # no v1 blobs -> finalize-ready
[[ "$output" == *"v2-only"* ]] || false # surfaced as v2-only coverage
}
```
- [ ] **Step 2: Run it, confirm it fails**
Run: `bats test/migrate.bats -f "counts v2-only externals"`
Expected: FAIL — `_migrate_status` currently only counts `*.gradle-properties.age`; it never mentions `v2-only`.
- [ ] **Step 3: Extend `_migrate_status`.** In `_migrate_status` (`secrets:2190`), inside the `for dir` loop, after the existing `while ... done < <(find "$dir" -type f -name '*.gradle-properties.age' ...)` block and before the per-project classification, add a v2-only count, then surface it in the per-project line. Concretely, replace the classification block:
```bash
if [ "$v1" -eq 0 ]; then
echo " $project: v2-ready (no v1 properties blobs)"
elif [ "$untwinned" -eq 0 ]; then
echo " $project: migrated ($v1 v1 blob(s), all twinned)"
else
echo " $project: NEEDS MIGRATE ($untwinned of $v1 v1 blob(s) un-twinned) — cd into the project and run 'secrets migrate'"
any_untwinned=1
fi
```
with:
```bash
# v2-only externals: a .properties.age with no .gradle-properties.age twin —
# old (v1) clients cannot read these (the additive-v2 forcing function).
local v2only=0 pf
while IFS= read -r pf; do
[ -f "$pf" ] || continue
[ -f "${pf%.properties.age}.gradle-properties.age" ] || v2only=$((v2only + 1))
done < <(find "$dir" -type f -name '*.properties.age' 2>/dev/null)
local v2note=""
[ "$v2only" -gt 0 ] && v2note=" [$v2only v2-only — old clients not served]"
if [ "$v1" -eq 0 ]; then
echo " $project: v2-ready (no v1 properties blobs)$v2note"
elif [ "$untwinned" -eq 0 ]; then
echo " $project: migrated ($v1 v1 blob(s), all twinned)$v2note"
else
echo " $project: NEEDS MIGRATE ($untwinned of $v1 v1 blob(s) un-twinned) — cd into the project and run 'secrets migrate'$v2note"
any_untwinned=1
fi
```
(`local` inside the loop is bash-3.2-fine — it re-declares per iteration.)
- [ ] **Step 4: Run the new test + status suite**
Run: `bats test/migrate.bats -f "status"`
Expected: all status tests PASS, including the new v2-only one.
- [ ] **Step 5: Run the full suite**
Run: `bats test/`
Expected: all pass.
- [ ] **Step 6: Commit**
```bash
git add secrets test/migrate.bats
git commit -m "feat: migrate --status surfaces v2-only externals (coverage, EGB-712)"
```
---
## Task 5: Docs, help, version, changelog
**Files:** `secrets` (`cmd_help`), `CLAUDE.md`, `README.md`, `VERSION`, `CHANGELOG.md`.
- [ ] **Step 1: `cmd_help` — reframe finalize as optional.** In `cmd_help()`, replace the finalize line:
```
secrets migrate --finalize Drop v1 blobs and mark the store v2 (after verify)
```
with:
```
secrets migrate --finalize Optional GC: drop v1 blobs and mark the store pure v2
```
- [ ] **Step 2: `CLAUDE.md` — additive-v2 paragraph.** In the "Store format" bullet (line ~65), after the migration-chain sentence, add (new sentence, same bullet):
```
Additive v2 (EGB-712): upgraded clients read either blob suffix (`_resolve_external_blob_read` tries `.properties.age` then `.gradle-properties.age`) and dual-write a `properties` external only when a v1 twin already exists (`_external_blob_write_targets`) — so existing externals keep old clients fresh, brand-new externals are v2-only (a gentle forcing function), and `migrate --finalize` is now OPTIONAL GC rather than a required, coordination-gated flag-day. dotenv and `file` blobs are identical across formats, so they always propagate to old clients.
```
- [ ] **Step 3: `README.md` — propagation table.** Add, near the migrate rows in the command table or in a short "Upgrading / store format" subsection, the propagation-by-secret-type summary (verbatim from spec §6):
```markdown
**Do teammates on an older `secrets` get new secrets?**
| Secret type | Old client gets it? |
|---|---|
| `.env` / `.env.*` / `.dev.vars` | **Yes, always** (blob name identical across formats) |
| whole-file external | **Yes, always** |
| `properties` external that already existed | **Yes** (dual-written) |
| brand-new `properties` external | **No — must upgrade `secrets`** (the forcing function) |
"Upgrade your secrets" = `git pull` the tool clone (binary ≥ 0.6.0.0) and/or `secrets migrate` the store. A read-only teammate only needs the tool `git pull`.
```
- [ ] **Step 4: `VERSION`** — set to `0.7.0.0`.
- [ ] **Step 5: `CHANGELOG.md`** — insert above the top entry:
```markdown
## [0.7.0.0] - 2026-06-08
### Changed
- **Additive store-format v2 (EGB-712)** — upgraded `secrets` clients now read
either external blob suffix (`.properties.age` or the legacy
`.gradle-properties.age`) and **dual-write** a `properties` external whenever a
v1 twin already exists in the store. Existing externals keep working for
teammates on an older `secrets`; only a brand-new `properties` external is
written v2-only (a gentle "upgrade to see it" forcing function). dotenv and
whole-`file` externals are unchanged across formats and always propagate.
- **`secrets migrate --finalize` is now optional GC**, not a required milestone.
Because clients dual-write and read-fall-back, no teammate is ever cut off by
*not* finalizing; finalize only reclaims the duplicate v1 blobs, and stays
deferrable indefinitely. Its safety gates are unchanged.
### Added
- **`secrets migrate --status`** now reports `v2-only` externals per project
(the ones an un-upgraded client cannot read), so you can see the forcing
function's footprint at a glance.
```
- [ ] **Step 6: Run the full suite**
Run: `bats test/`
Expected: all pass (docs don't affect tests). Confirm the count is baseline 242 + net new tests from Tasks 1/2/4 (read-fallback, two write tests, v2-only status) minus the rewritten back-compat test (replaced, not added) = **246**.
- [ ] **Step 7: Sanity — help renders**
Run: `./secrets help 2>&1 | grep -- "--finalize"`
Expected: shows the reframed "Optional GC" line. (Help only prints; touches no store.)
- [ ] **Step 8: Commit**
```bash
git add secrets CLAUDE.md README.md VERSION CHANGELOG.md
git commit -m "docs: additive-v2 propagation + optional-GC finalize; bump 0.7.0.0 (EGB-712)"
```
---
## Self-review against the spec
- **§1 read resolution** → Task 1 (`_resolve_external_blob_read`, wired into pull + verify). Test: read-fallback.
- **§2 write rule / twin rule** → Task 2 (`_external_blob_write_targets`, push wiring). Tests: fresh-→v2-only, existing-twin-→dual-write.
- **§3 marker NOT stamped** → no production change (push never touches the marker); guarded implicitly by Task 3 keeping the `make_v1_store; push; migrate` flow working (store stays markerless after push). The plan deliberately does not add marker-stamping.
- **§4 finalize = optional GC** → unchanged logic; reframed in Task 5 docs/help. Existing finalize tests stay green (Task 3 keeps them green).
- **§5 migrate / --status coverage** → Task 4 (v2-only line); `migrate` copy-forward unchanged (EGB-710).
- **§6 propagation table** → Task 5 README/CLAUDE.md.
- **§7 caveat** → documented in CHANGELOG/CLAUDE.md framing; the read-fallback test exercises the v1-only read path.
- **§8 back-compat matrix** → covered across Task 1 (reads), Task 2 (writes), Task 3 (old-client v1 blobs simulated).
- **Safety invariants** → no path-rail changes; bash 3.2 (no associative arrays; `while read` + `find`); recursive walks unchanged; finalize gating untouched.
**Type/name consistency:** `_resolve_external_blob_read(project, slug, mtype)` and `_external_blob_write_targets(project, slug, mtype)` use the same arg order everywhere; test helpers `m_fake_v1_twin` (keep both) and `m_make_v1_only` (rename to v1-only) are used consistently per their documented semantics.
**Placeholder scan:** none — every step carries verbatim code or an exact enumerated edit with the precise insertion point.
## Operator-local follow-up (not part of this plan)
Per `.ship-policy.json`, before any PR ask the operator to run `./test/run-security.sh` and complete the SIGNOFF. EGB-712 also needs its one stale AC bullet ("Marker auto-stamps…") corrected to match the §3 decision (no auto-stamp) — a one-line Linear edit.

View file

@ -0,0 +1,216 @@
# Additive v2 — automatic dual-write (no flag-day finalize)
**Status:** Design approved (2026-06-08), pending implementation plan.
**Supersedes the framing of:** EGB-703 ("finalize is the destructive milestone you must reach") and EGB-709 ("delete v1 paths once every client is v2").
**Relates to:** EGB-677 (storage-model unification), EGB-710 (manifest-free migrate).
## Problem
Store-format v2 (EGB-703) renamed the `properties` external blob suffix
(`.gradle-properties.age``.properties.age`) and made the transition a
three-step migration ending in a **destructive** `secrets migrate --finalize`
that drops the v1 blobs. Finalize is gated on "every machine and store in use is
v2" — a coordination requirement that's trivial for a solo dev but **uncertain
for a distributed team of 23+**: there is no reliable "all-clear" signal, and
finalizing early silently cuts an un-upgraded client off from `properties`
externals (it looks for `.gradle-properties.age`, which is gone).
The hard gate exists **only** because finalize permanently drops the v1 blobs.
Remove the obligation to drop, and there is nothing to coordinate.
## Decision
Adopt **Strategy B — Additive v2**: "becoming v2" stops being a destructive
milestone and becomes a property an upgraded client maintains **automatically**.
Reads try both suffixes; writes keep alive whatever old clients already knew.
There is no required `migrate` ceremony and no required `finalize` — **upgrading
the `secrets` tool *is* the migration.** `finalize` survives only as optional,
indefinitely-deferrable garbage collection.
Direction chosen over the alternatives: "park it / never finalize" (leaves the
two-format wart in place, resolves nothing) and "abandon the suffix rename"
(reverts shipped EGB-703 behavior, keeps the ugly `gradle-properties` suffix
forever). Additive v2 is the only option that reaches a clean v2 end-state
*without* a coordinated flag-day.
## Design
### 1. Read resolution — suffix-agnostic
Any upgraded client resolving a `properties` external blob looks for the v2
suffix `.properties.age` first, then falls back to the v1 `.gradle-properties.age`.
An upgraded client therefore **never fails to find a blob** regardless of which
suffix is on disk. Old (pre-0.6.0.0) clients still read v1-only — shipped code
cannot be changed.
Code: today's `_external_blob_suffix(type)` (a single suffix string) is replaced
by a resolver that returns the **path of the blob that exists** for a given
`(project, slug, type)`, trying v2 then v1. `file` externals are unchanged in
both formats, so the resolver is a no-op identity for them (single suffix
`.file.age`).
### 2. Write rule — the self-managing twin rule
On push of a `properties` external, the client checks the store for an existing
v1 twin (`<slug>.gradle-properties.age`):
- **v1 twin exists** (the external was first pushed by an old client) →
**dual-write both suffixes.** Old clients stay fresh forever; nobody is cut off.
- **No v1 twin exists** (brand-new external, first pushed by an upgraded client) →
**write v2-only (`.properties.age`).** Old clients cannot see it → the gentle,
*intended* forcing function. It only ever bites on genuinely new externals,
never on anything that previously worked.
No stored state is required: the presence/absence of the v1 twin **is** the
signal. Mental model: *"keep alive what old clients already knew; new things are
v2-only."*
### 3. The marker — unchanged meaning, NOT auto-stamped
Additive-v2 read/write behavior does **not** depend on the `.secrets-format`
marker at all (reads try both suffixes; writes follow the twin rule). So the
marker is left exactly as it is today: set only by `init` (born-v2, a fresh pure-v2
store) or `migrate --finalize` (a store GC'd to pure v2). A transitional store
that upgraded clients are dual-writing stays **markerless (v1)** — which is
*accurate*: it has not been finalized to pure v2, and v1 twins still exist.
Push deliberately does **not** auto-stamp the marker. (An earlier draft proposed
auto-stamping on first push; that was dropped during planning because it would
make a v1 store read as v2 the moment anyone pushed — turning every `migrate`
into a no-op and contradicting the "v1 until finalized" model the whole
migration relies on. The marker's only purposes — `secrets which` display and
gating `migrate`/`finalize` — are better served by it continuing to mean
"finalized/pure v2.")
`secrets which` continues to report `format: vN` from the marker.
### 4. `finalize` → optional GC, never required
`secrets migrate --finalize` remains the *only* operation that stops dual-writing
and drops the v1 twins. It is pure space reclamation (kilobytes), still
coordination-gated **if** you choose to run it, but never obligatory and
deferrable forever. Its existing safety posture is unchanged (recovery tag,
`verify --all` green gate, twin-before-drop, `--yes`/operator confirmation). This
is what defuses the gate: the destructive step still exists but is now optional
housekeeping, not a release blocker.
### 5. `migrate` / `migrate --status` → normalize + inspect
- `secrets migrate` (the EGB-710 manifest-free copy-forward) stays as an optional
"backfill v2 twins for existing v1-only externals" command — useful right
before a `finalize`. With read-fallback (§1) it is no longer required for
correctness, only for tidiness.
- `secrets migrate --status` becomes the **coverage survey**: per external, which
suffixes exist, and whether old clients are still being served (i.e. whether a
v1 twin is still present and being dual-written). This is the operator's
dashboard for "is anyone still relying on v1?" before an optional `finalize`.
### 6. Propagation semantics (→ user docs verbatim)
"Has not migrated" means **old client** (`secrets` < 0.6.0.0), not "hasn't run
`migrate`". `migrate` is a per-store op done once by anyone; what protects a
teammate is their **client version**. A read-only teammate needs only the tool
`git pull` (binary ≥ 0.6.0.0), not to run `migrate` themselves. "Upgrade your
secrets" (git pull the tool and/or migrate the store) is the correct umbrella —
and for a solo dev the two are one motion.
Does a teammate on an old client get a secret User 1 just added?
| Secret type | Blob name v1 vs v2 | Old client gets the new secret? |
|---|---|---|
| **dotenv** (`.env`, `.env.*`, `.dev.vars`) | identical (`.env.age`) | **Yes, always.** No forcing function possible — the blob name never changed. |
| **`file` external** (keystore, etc.) | identical (`.file.age`) | **Yes, always.** |
| **`properties` external** — existing (has a v1 twin) | dual-written | **Yes.** Old client reads the maintained `.gradle-properties.age`. |
| **`properties` external** — brand-new (no v1 twin) | v2-only | **No → must upgrade.** The forcing function; rare, and never breaks anything that previously worked. |
Net: a teammate on an old client keeps getting **all** everyday `.env` updates
indefinitely (a good safety property — no one silently misses everyday secrets),
and only hits a wall on a genuinely new `properties`-style external.
### 7. Known caveat (documented, not engineered around)
Dual-write keeps v1 *readers* fresh, but a v1 *writer* writes only
`.gradle-properties.age`. A v2 reader (reading `.properties.age` first) could
therefore read stale data until that external is re-pushed by an upgraded client.
For the normal shape — one writer per external, who upgrades first — it never
bites. This matches today's reality and is documented rather than solved
(solving it would require timestamp/newest-wins arbitration across two encrypted
blobs — YAGNI for v1).
### 8. Backward-compatibility matrix
| Actor | dotenv / `file` | `properties` (existing twin) | `properties` (new, v2-only) |
|---|---|---|---|
| Upgraded client reads | ✓ | ✓ (resolver finds either) | ✓ |
| Upgraded client writes | unchanged | dual-writes both | writes v2-only |
| Old client reads | ✓ | ✓ (reads maintained v1 twin) | ✗ (forcing function) |
| Old client writes | unchanged | writes v1 twin only (see §7) | n/a (can't create v2) |
## Code-level surface (for the implementation plan)
- **`_external_blob_suffix(type)`** → split into:
- `_resolve_external_blob_read(project, slug, type)` — returns the path of the
blob that exists, trying `.properties.age` then `.gradle-properties.age` for
`properties`; identity for `file`. Used by `pull_external_files`, `verify`,
and any read path.
- `_external_blob_write_targets(project, slug, type)` — returns the suffix
path(s) to write: for `properties`, both suffixes when a v1 twin already
exists, else v2-only; single path for `file`.
- **`push_external_files`** — write to every path from
`_external_blob_write_targets` (was a single `age -o`). Push does **not** stamp
the marker (see §3).
- **`pull_external_files` / `cmd_verify`** — resolve blobs via
`_resolve_external_blob_read` (was the single-suffix lookup).
- **`_migrate_finalize`** — unchanged logic; doc/help reframed as optional GC.
- **`_migrate_project` / `_migrate_status`** — retained (EGB-710); `--status`
extended to report dual-write coverage per external.
- **Docs:** CLAUDE.md "Store format" bullet, README, `cmd_help`, CHANGELOG,
VERSION bump (minor — new write semantics).
## Safety / constraints (unchanged invariants)
- bash 3.2 portable; every store walk stays recursive (`find -type f`).
- Path-validation rails (`_validate_external_target_path`, slug derivation,
symlink/`..` refusal) untouched.
- `finalize`'s destructive gating (recovery tag, verify-green, twin-before-drop,
`--yes`) untouched.
- The store still carries no manifest; `.secrets.json` remains per-project.
## Test plan (bats, outline)
1. Read-fallback: a v2 client resolves a `properties` external that exists only
as `.gradle-properties.age` (no twin) — pull succeeds.
2. Twin rule — existing twin → dual-write: push an external that has a v1 twin;
assert **both** suffixes are written and an old-client read path (v1 suffix)
sees the fresh value.
3. Twin rule — new external → v2-only: push a brand-new `properties` external;
assert **only** `.properties.age` is written (no v1 twin created) — the
forcing function.
4. Marker NOT stamped on push: a `make_v1_store` + push leaves `.secrets-format`
absent (store stays v1/transitional); the existing `make_v1_store; push;
migrate` flow is unaffected.
5. dotenv/`file` unaffected: a dotenv and a `file` external round-trip identically
regardless of store marker.
6. `migrate --status` coverage: reports which externals are dual-written vs
v2-only.
7. `finalize` still green: existing finalize gates and drop behavior unchanged.
8. Caveat is observable (optional): a v1-suffix-only update is read by the v2
client via fallback (documents the one-writer assumption).
## Ripple to the roadmap
- **EGB-703**: "finalize is the destructive milestone you must reach" → "finalize
is optional GC." Update the ticket/notes.
- **EGB-709** (collapse v1 paths): no longer gates on "all clients v2." Becomes
"delete the dual-write/transitional code **if/when** every store is GC'd to
pure v2" — much later, low stakes. Add a note to EGB-709.
- A new ticket should track this work (additive-v2 dual-write).
## Decided knobs (no longer open)
- Migration ceremony: **automatic** (upgraded client dual-writes on push; no
required `migrate`). `migrate`/`--status` remain manual/inspection tools.
- Forcing function: **kept**, scoped to brand-new `properties` externals via the
twin rule (existing twins always dual-written).
- `finalize`: **kept** as optional GC (not removed), so a fully-upgraded store
can still be reclaimed to pure v2.

96
secrets
View file

@ -539,19 +539,49 @@ _store_format() {
echo 1 echo 1
} }
# The on-disk blob suffix for an external entry, format-aware. v2 unifies # Resolve the on-disk path of an external blob for READING. Tries the v2 suffix
# the legacy `gradle-properties` suffix to `properties` (matching the JSON # (.properties.age) first, then falls back to the v1 (.gradle-properties.age) for
# manifest `type`); `file` is unchanged in both formats. The slug + this # `properties` externals, so an upgraded client finds the blob whichever format
# suffix + `.age` is the external blob name. This is the single source of # wrote it (additive v2 — EGB-712). `file` externals share one suffix in both
# truth for the suffix — push, pull, verify all route through it so a v1 # formats. Echoes the path of the blob that exists; if neither exists, echoes the
# and a v2 store can never disagree on where a blob lives. # canonical v2 path so the caller's "no blob" message reads sensibly. Read-only.
_external_blob_suffix() { _resolve_external_blob_read() {
local mtype="$1" local project="$1" slug="$2" mtype="$3"
if [ "$mtype" = "gradle-properties" ] && [ "$(_store_format)" = "2" ]; then local base="$SECRETS_DIR/$project/external/$slug"
echo "properties" case "$mtype" in
file)
echo "$base.file.age" ;;
properties|gradle-properties)
if [ -f "$base.properties.age" ]; then
echo "$base.properties.age"
elif [ -f "$base.gradle-properties.age" ]; then
echo "$base.gradle-properties.age"
else else
echo "$mtype" echo "$base.properties.age"
fi fi ;;
*)
echo "$base.$mtype.age" ;;
esac
}
# The on-disk path(s) to WRITE for an external blob, one per line. For a
# `properties` external this is the v2 suffix (.properties.age) ALWAYS, plus the
# v1 suffix (.gradle-properties.age) WHEN a v1 twin already exists in the store
# (dual-write keeps old clients fresh; a brand-new external is v2-only — the
# intended forcing function, additive v2 / EGB-712). `file` externals have a
# single suffix in both formats. Independent of the store marker.
_external_blob_write_targets() {
local project="$1" slug="$2" mtype="$3"
local base="$SECRETS_DIR/$project/external/$slug"
case "$mtype" in
file)
echo "$base.file.age" ;;
properties|gradle-properties)
echo "$base.properties.age"
[ -f "$base.gradle-properties.age" ] && echo "$base.gradle-properties.age" ;;
*)
echo "$base.$mtype.age" ;;
esac
} }
# Merge managed key=value lines (from $2) into target file $1, preserving # Merge managed key=value lines (from $2) into target file $1, preserving
@ -652,7 +682,11 @@ push_external_files() {
# EGB-652: whole-file sync — encrypt the file verbatim (binary-safe). # EGB-652: whole-file sync — encrypt the file verbatim (binary-safe).
mkdir -p "$SECRETS_DIR/$project/external" mkdir -p "$SECRETS_DIR/$project/external"
local fslug; fslug=$(_secrets_files_slug "$mpath") local fslug; fslug=$(_secrets_files_slug "$mpath")
age -r "$pubkey" -o "$SECRETS_DIR/$project/external/$fslug.$(_external_blob_suffix file).age" "$expanded" local wt
while IFS= read -r wt; do
[ -n "$wt" ] || continue
age -r "$pubkey" -o "$wt" "$expanded"
done < <(_external_blob_write_targets "$project" "$fslug" file)
info "Encrypted file $mpath" info "Encrypted file $mpath"
pushed=$((pushed + 1)) pushed=$((pushed + 1))
continue continue
@ -681,7 +715,11 @@ push_external_files() {
fi fi
mkdir -p "$SECRETS_DIR/$project/external" mkdir -p "$SECRETS_DIR/$project/external"
local slug; slug=$(_secrets_files_slug "$mpath") local slug; slug=$(_secrets_files_slug "$mpath")
age -r "$pubkey" -o "$SECRETS_DIR/$project/external/$slug.$(_external_blob_suffix "$mtype").age" "$tmp" local wt
while IFS= read -r wt; do
[ -n "$wt" ] || continue
age -r "$pubkey" -o "$wt" "$tmp"
done < <(_external_blob_write_targets "$project" "$slug" "$mtype")
rm -f "$tmp" rm -f "$tmp"
info "Extracted $found key(s) from $mpath" info "Extracted $found key(s) from $mpath"
pushed=$((pushed + 1)) pushed=$((pushed + 1))
@ -710,7 +748,7 @@ pull_external_files() {
continue continue
fi fi
local slug; slug=$(_secrets_files_slug "$mpath") local slug; slug=$(_secrets_files_slug "$mpath")
local blob="$SECRETS_DIR/$project/external/$slug.$(_external_blob_suffix "$mtype").age" local blob; blob=$(_resolve_external_blob_read "$project" "$slug" "$mtype")
if [ ! -f "$blob" ]; then if [ ! -f "$blob" ]; then
echo "WARNING: $SECRETS_FILES_NAME names '$mpath' but no encrypted data exists in the store yet. Run 'secrets push' on a machine that has these keys. Skipping." >&2 echo "WARNING: $SECRETS_FILES_NAME names '$mpath' but no encrypted data exists in the store yet. Run 'secrets push' on a machine that has these keys. Skipping." >&2
continue continue
@ -2053,14 +2091,13 @@ _verify_project() {
while IFS=$'\t' read -r etype epath _; do while IFS=$'\t' read -r etype epath _; do
[ -n "$etype" ] || continue [ -n "$etype" ] || continue
slug=$(_secrets_files_slug "$epath") slug=$(_secrets_files_slug "$epath")
erel="external/$slug.$(_external_blob_suffix "$etype").age" # Account for BOTH suffix forms in the orphan set — a dual-written `properties`
# Account for BOTH the v1 and v2 suffix forms in the orphan set. During the # external (additive v2 — EGB-712) legitimately has both blobs on disk; neither
# migration window (after copy-forward, before --finalize) the v2 twin # is an orphan. (file's two forms are identical.)
# coexists with the v1 blob; neither should read as an orphan whichever
# format the store currently reports. (file's two forms are identical.)
expected="${expected}external/$slug.$etype.age"$'\n' expected="${expected}external/$slug.$etype.age"$'\n'
[ "$etype" = "gradle-properties" ] && expected="${expected}external/$slug.properties.age"$'\n' [ "$etype" = "gradle-properties" ] && expected="${expected}external/$slug.properties.age"$'\n'
eblob="$pdir/$erel" eblob=$(_resolve_external_blob_read "$project" "$slug" "$etype")
erel="${eblob#"$pdir"/}"
if [ ! -f "$eblob" ]; then if [ ! -f "$eblob" ]; then
echo "FINDING: external '$epath' ($etype) is declared but has no blob in the store ($project/$erel missing). Run 'secrets push'." >&2 echo "FINDING: external '$epath' ($etype) is declared but has no blob in the store ($project/$erel missing). Run 'secrets push'." >&2
findings=$((findings + 1)) findings=$((findings + 1))
@ -2205,12 +2242,21 @@ _migrate_status() {
new="${f%.gradle-properties.age}.properties.age" new="${f%.gradle-properties.age}.properties.age"
[ -f "$new" ] || untwinned=$((untwinned + 1)) [ -f "$new" ] || untwinned=$((untwinned + 1))
done < <(find "$dir" -type f -name '*.gradle-properties.age' 2>/dev/null) done < <(find "$dir" -type f -name '*.gradle-properties.age' 2>/dev/null)
# v2-only externals: a .properties.age with no .gradle-properties.age twin —
# old (v1) clients cannot read these (the additive-v2 forcing function).
local v2only=0 pf
while IFS= read -r pf; do
[ -f "$pf" ] || continue
[ -f "${pf%.properties.age}.gradle-properties.age" ] || v2only=$((v2only + 1))
done < <(find "$dir" -type f -name '*.properties.age' 2>/dev/null)
local v2note=""
[ "$v2only" -gt 0 ] && v2note=" [$v2only v2-only — old clients not served]"
if [ "$v1" -eq 0 ]; then if [ "$v1" -eq 0 ]; then
echo " $project: v2-ready (no v1 properties blobs)" echo " $project: v2-ready (no v1 properties blobs)$v2note"
elif [ "$untwinned" -eq 0 ]; then elif [ "$untwinned" -eq 0 ]; then
echo " $project: migrated ($v1 v1 blob(s), all twinned)" echo " $project: migrated ($v1 v1 blob(s), all twinned)$v2note"
else else
echo " $project: NEEDS MIGRATE ($untwinned of $v1 v1 blob(s) un-twinned) — cd into the project and run 'secrets migrate'" echo " $project: NEEDS MIGRATE ($untwinned of $v1 v1 blob(s) un-twinned) — cd into the project and run 'secrets migrate'$v2note"
any_untwinned=1 any_untwinned=1
fi fi
done done
@ -2343,7 +2389,7 @@ Usage:
secrets verify --all Decrypt-test every blob in every project (integrity gate) secrets verify --all Decrypt-test every blob in every project (integrity gate)
secrets migrate [--dry-run] Copy-forward this project's v1 blobs to store format v2 secrets migrate [--dry-run] Copy-forward this project's v1 blobs to store format v2
secrets migrate --status Survey every project's v2 readiness (finalize gate) secrets migrate --status Survey every project's v2 readiness (finalize gate)
secrets migrate --finalize Drop v1 blobs and mark the store v2 (after verify) secrets migrate --finalize Optional GC: drop v1 blobs and mark the store pure v2
secrets which Show the active store, manifest, and external entries secrets which Show the active store, manifest, and external entries
secrets where Alias for `which` secrets where Alias for `which`
secrets status Alias for `which` secrets status Alias for `which`

View file

@ -10,6 +10,19 @@ make_v1_store() {
init_with_remote init_with_remote
rm -f "$SECRETS_DIR/.secrets-format" rm -f "$SECRETS_DIR/.secrets-format"
} }
# Simulate an old (v1) client's properties blob: copy the pushed v2
# .properties.age to its v1 .gradle-properties.age twin (KEEPS both present).
m_fake_v1_twin() {
local proj="$1" v2
v2=$(ls "$SECRETS_DIR/$proj/external/"*.properties.age)
cp "$v2" "${v2%.properties.age}.gradle-properties.age"
}
# Like m_fake_v1_twin but RENAMES (leaves ONLY the v1 blob) — for copy-forward fixtures.
m_make_v1_only() {
local proj="$1" v2
v2=$(ls "$SECRETS_DIR/$proj/external/"*.properties.age)
mv "$v2" "${v2%.properties.age}.gradle-properties.age"
}
m_gradle_src() { mkdir -p "$HOME/.gradle"; printf '%s' "$1" > "$HOME/.gradle/gradle.properties"; } m_gradle_src() { mkdir -p "$HOME/.gradle"; printf '%s' "$1" > "$HOME/.gradle/gradle.properties"; }
m_file_src() { mkdir -p "$HOME/keystores"; printf 'KS\x00\x01\x02\xffDATA\n' > "$HOME/keystores/upload.keystore"; } m_file_src() { mkdir -p "$HOME/keystores"; printf 'KS\x00\x01\x02\xffDATA\n' > "$HOME/keystores/upload.keystore"; }
@ -49,13 +62,41 @@ m_file_src() { mkdir -p "$HOME/keystores"; printf 'KS\x00\x01\x02\xffDATA\n' >
[ "$status" -ne 0 ] [ "$status" -ne 0 ]
} }
@test "push on a v1 store still writes .gradle-properties.age (back-compat)" { @test "push on a v1 store writes the v2 suffix for a fresh external (additive v2)" {
make_v1_store make_v1_store
m_gradle_src $'beaconClerkPkTest=pk_test_abc\n' m_gradle_src $'beaconClerkPkTest=pk_test_abc\n'
create_project_dir v1push create_project_dir v1push
printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files
"$SECRETS_BIN" push v1push >/dev/null 2>&1 "$SECRETS_BIN" push v1push >/dev/null 2>&1
run bash -c "ls $SECRETS_DIR/v1push/external/*.gradle-properties.age" run bash -c "ls $SECRETS_DIR/v1push/external/*.properties.age"
[ "$status" -eq 0 ]
}
@test "push writes the v2 suffix for a fresh external even on a v1 store" {
make_v1_store
m_gradle_src $'beaconClerkPkTest=pk_test_abc\n'
create_project_dir freshv1
printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files
"$SECRETS_BIN" push freshv1 >/dev/null 2>&1
run bash -c "ls $SECRETS_DIR/freshv1/external/*.properties.age"
[ "$status" -eq 0 ]
run bash -c "ls $SECRETS_DIR/freshv1/external/*.gradle-properties.age 2>/dev/null"
[ "$status" -ne 0 ]
}
@test "push dual-writes the v1 twin so old clients stay fresh" {
make_v1_store
m_gradle_src $'beaconClerkPkTest=pk_test_old\n'
create_project_dir dualwrite
printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files
"$SECRETS_BIN" push dualwrite >/dev/null 2>&1
m_fake_v1_twin dualwrite
m_gradle_src $'beaconClerkPkTest=pk_test_new\n'
"$SECRETS_BIN" push dualwrite >/dev/null 2>&1
rm -f "$SECRETS_DIR/dualwrite/external/"*.properties.age
rm -f "$HOME/.gradle/gradle.properties"
"$SECRETS_BIN" pull dualwrite >/dev/null 2>&1
run grep -q 'beaconClerkPkTest=pk_test_new' "$HOME/.gradle/gradle.properties"
[ "$status" -eq 0 ] [ "$status" -eq 0 ]
} }
@ -67,6 +108,21 @@ m_file_src() { mkdir -p "$HOME/keystores"; printf 'KS\x00\x01\x02\xffDATA\n' >
[ "$status" -eq 0 ] [ "$status" -eq 0 ]
} }
@test "pull reads a v1-suffix properties blob on a v2 store (read-fallback)" {
init_with_remote # born-v2 store (marker=2)
m_gradle_src $'beaconClerkPkTest=pk_test_v1\n'
create_project_dir rffallback
printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files
"$SECRETS_BIN" push rffallback >/dev/null 2>&1 # writes .properties.age on a v2 store
# Simulate an external that exists only in the v1 suffix (an old client wrote it):
local v2blob; v2blob=$(ls "$SECRETS_DIR/rffallback/external/"*.properties.age)
mv "$v2blob" "${v2blob%.properties.age}.gradle-properties.age"
rm -f "$HOME/.gradle/gradle.properties"
"$SECRETS_BIN" pull rffallback >/dev/null 2>&1
run grep -q 'beaconClerkPkTest=pk_test_v1' "$HOME/.gradle/gradle.properties"
[ "$status" -eq 0 ]
}
# ─── migrate --dry-run / copy-forward (increment 2) ─────────────────── # ─── migrate --dry-run / copy-forward (increment 2) ───────────────────
@test "migrate --dry-run reports the rename and writes nothing" { @test "migrate --dry-run reports the rename and writes nothing" {
@ -75,6 +131,7 @@ m_file_src() { mkdir -p "$HOME/keystores"; printf 'KS\x00\x01\x02\xffDATA\n' >
create_project_dir dryproj create_project_dir dryproj
printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files
"$SECRETS_BIN" push dryproj >/dev/null 2>&1 "$SECRETS_BIN" push dryproj >/dev/null 2>&1
m_make_v1_only dryproj
run "$SECRETS_BIN" migrate --dry-run run "$SECRETS_BIN" migrate --dry-run
[ "$status" -eq 0 ] [ "$status" -eq 0 ]
[[ "$output" == *"would migrate"* ]] || false [[ "$output" == *"would migrate"* ]] || false
@ -100,6 +157,7 @@ m_file_src() { mkdir -p "$HOME/keystores"; printf 'KS\x00\x01\x02\xffDATA\n' >
create_project_dir cfproj create_project_dir cfproj
printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files
"$SECRETS_BIN" push cfproj >/dev/null 2>&1 "$SECRETS_BIN" push cfproj >/dev/null 2>&1
m_make_v1_only cfproj
local old; old=$(ls "$SECRETS_DIR/cfproj/external/"*.gradle-properties.age) local old; old=$(ls "$SECRETS_DIR/cfproj/external/"*.gradle-properties.age)
run "$SECRETS_BIN" migrate run "$SECRETS_BIN" migrate
[ "$status" -eq 0 ] [ "$status" -eq 0 ]
@ -115,6 +173,7 @@ m_file_src() { mkdir -p "$HOME/keystores"; printf 'KS\x00\x01\x02\xffDATA\n' >
create_project_dir idemproj create_project_dir idemproj
printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files
"$SECRETS_BIN" push idemproj >/dev/null 2>&1 "$SECRETS_BIN" push idemproj >/dev/null 2>&1
m_make_v1_only idemproj
"$SECRETS_BIN" migrate >/dev/null 2>&1 "$SECRETS_BIN" migrate >/dev/null 2>&1
run "$SECRETS_BIN" migrate run "$SECRETS_BIN" migrate
[ "$status" -eq 0 ] [ "$status" -eq 0 ]
@ -131,6 +190,7 @@ m_file_src() { mkdir -p "$HOME/keystores"; printf 'KS\x00\x01\x02\xffDATA\n' >
create_project_dir nomanifestblob create_project_dir nomanifestblob
printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files
"$SECRETS_BIN" push nomanifestblob >/dev/null 2>&1 "$SECRETS_BIN" push nomanifestblob >/dev/null 2>&1
m_make_v1_only nomanifestblob
rm -f .secrets.json # simulate a pre-manifest project rm -f .secrets.json # simulate a pre-manifest project
run "$SECRETS_BIN" migrate run "$SECRETS_BIN" migrate
[ "$status" -eq 0 ] [ "$status" -eq 0 ]
@ -146,6 +206,7 @@ m_file_src() { mkdir -p "$HOME/keystores"; printf 'KS\x00\x01\x02\xffDATA\n' >
create_project_dir staleblob create_project_dir staleblob
printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files
"$SECRETS_BIN" push staleblob >/dev/null 2>&1 "$SECRETS_BIN" push staleblob >/dev/null 2>&1
m_make_v1_only staleblob
# The blob is now in the store. Drop the external from the project's manifest # The blob is now in the store. Drop the external from the project's manifest
# entirely (and remove the legacy file) so NO manifest declares it. # entirely (and remove the legacy file) so NO manifest declares it.
printf '{"version":2,"dotenv":[".env",".env.staging"]}\n' > .secrets.json printf '{"version":2,"dotenv":[".env",".env.staging"]}\n' > .secrets.json
@ -206,6 +267,7 @@ m_file_src() { mkdir -p "$HOME/keystores"; printf 'KS\x00\x01\x02\xffDATA\n' >
create_project_dir failverify create_project_dir failverify
printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files
"$SECRETS_BIN" push failverify >/dev/null 2>&1 "$SECRETS_BIN" push failverify >/dev/null 2>&1
m_make_v1_only failverify
"$SECRETS_BIN" migrate >/dev/null 2>&1 "$SECRETS_BIN" migrate >/dev/null 2>&1
# corrupt the v2 twin so verify --all fails # corrupt the v2 twin so verify --all fails
printf 'garbage' > "$SECRETS_DIR/failverify/external/"*.properties.age printf 'garbage' > "$SECRETS_DIR/failverify/external/"*.properties.age
@ -224,6 +286,7 @@ m_file_src() { mkdir -p "$HOME/keystores"; printf 'KS\x00\x01\x02\xffDATA\n' >
create_project_dir untwinned create_project_dir untwinned
printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files
"$SECRETS_BIN" push untwinned >/dev/null 2>&1 "$SECRETS_BIN" push untwinned >/dev/null 2>&1
m_make_v1_only untwinned
# do NOT migrate — leave the v1 blob with no twin # do NOT migrate — leave the v1 blob with no twin
run "$SECRETS_BIN" migrate --finalize --yes run "$SECRETS_BIN" migrate --finalize --yes
[ "$status" -eq 1 ] [ "$status" -eq 1 ]
@ -254,6 +317,7 @@ m_file_src() { mkdir -p "$HOME/keystores"; printf 'KS\x00\x01\x02\xffDATA\n' >
create_project_dir tagproj create_project_dir tagproj
printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files
"$SECRETS_BIN" push tagproj >/dev/null 2>&1 "$SECRETS_BIN" push tagproj >/dev/null 2>&1
m_make_v1_only tagproj
"$SECRETS_BIN" migrate >/dev/null 2>&1 "$SECRETS_BIN" migrate >/dev/null 2>&1
"$SECRETS_BIN" migrate --finalize --yes >/dev/null 2>&1 "$SECRETS_BIN" migrate --finalize --yes >/dev/null 2>&1
local tag; tag=$(git -C "$SECRETS_DIR" tag | grep '^pre-v2-migrate-') local tag; tag=$(git -C "$SECRETS_DIR" tag | grep '^pre-v2-migrate-')
@ -269,6 +333,7 @@ m_file_src() { mkdir -p "$HOME/keystores"; printf 'KS\x00\x01\x02\xffDATA\n' >
create_project_dir confproj create_project_dir confproj
printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files
"$SECRETS_BIN" push confproj >/dev/null 2>&1 "$SECRETS_BIN" push confproj >/dev/null 2>&1
m_make_v1_only confproj
"$SECRETS_BIN" migrate >/dev/null 2>&1 "$SECRETS_BIN" migrate >/dev/null 2>&1
run bash -c "echo '' | $SECRETS_BIN migrate --finalize" run bash -c "echo '' | $SECRETS_BIN migrate --finalize"
[ "$status" -eq 1 ] [ "$status" -eq 1 ]
@ -352,6 +417,7 @@ m_file_src() { mkdir -p "$HOME/keystores"; printf 'KS\x00\x01\x02\xffDATA\n' >
create_project_dir needsmig create_project_dir needsmig
printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files
"$SECRETS_BIN" push needsmig >/dev/null 2>&1 # v1 blob, no twin yet "$SECRETS_BIN" push needsmig >/dev/null 2>&1 # v1 blob, no twin yet
m_make_v1_only needsmig
run "$SECRETS_BIN" migrate --status run "$SECRETS_BIN" migrate --status
[ "$status" -ne 0 ] # not finalize-ready [ "$status" -ne 0 ] # not finalize-ready
[[ "$output" == *"needsmig"* ]] || false [[ "$output" == *"needsmig"* ]] || false
@ -371,6 +437,17 @@ m_file_src() { mkdir -p "$HOME/keystores"; printf 'KS\x00\x01\x02\xffDATA\n' >
[[ "$output" == *"Finalize-ready"* ]] || false [[ "$output" == *"Finalize-ready"* ]] || false
} }
@test "migrate --status counts v2-only externals (old clients not served)" {
make_v1_store
m_gradle_src $'beaconClerkPkTest=pk_test_abc\n'
create_project_dir v2onlyext
printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files
"$SECRETS_BIN" push v2onlyext >/dev/null 2>&1 # v2-only (fresh, no v1 twin)
run "$SECRETS_BIN" migrate --status
[ "$status" -eq 0 ] # no v1 blobs -> finalize-ready
[[ "$output" == *"v2-only"* ]] || false # surfaced as v2-only coverage
}
@test "migrate --status on an already-v2 store says nothing to do" { @test "migrate --status on an already-v2 store says nothing to do" {
init_with_remote init_with_remote
create_project_dir v2status create_project_dir v2status