diff --git a/docs/superpowers/specs/2026-06-08-additive-v2-dual-write-design.md b/docs/superpowers/specs/2026-06-08-additive-v2-dual-write-design.md new file mode 100644 index 0000000..7827e44 --- /dev/null +++ b/docs/superpowers/specs/2026-06-08-additive-v2-dual-write-design.md @@ -0,0 +1,203 @@ +# Additive v2 — automatic dual-write (no flag-day finalize) + +**Status:** Design approved (2026-06-08), pending implementation plan. +**Supersedes the framing of:** EGB-703 ("finalize is the destructive milestone you must reach") and EGB-709 ("delete v1 paths once every client is v2"). +**Relates to:** EGB-677 (storage-model unification), EGB-710 (manifest-free migrate). + +## Problem + +Store-format v2 (EGB-703) renamed the `properties` external blob suffix +(`.gradle-properties.age` → `.properties.age`) and made the transition a +three-step migration ending in a **destructive** `secrets migrate --finalize` +that drops the v1 blobs. Finalize is gated on "every machine and store in use is +v2" — a coordination requirement that's trivial for a solo dev but **uncertain +for a distributed team of 2–3+**: there is no reliable "all-clear" signal, and +finalizing early silently cuts an un-upgraded client off from `properties` +externals (it looks for `.gradle-properties.age`, which is gone). + +The hard gate exists **only** because finalize permanently drops the v1 blobs. +Remove the obligation to drop, and there is nothing to coordinate. + +## Decision + +Adopt **Strategy B — Additive v2**: "becoming v2" stops being a destructive +milestone and becomes a property an upgraded client maintains **automatically**. +Reads try both suffixes; writes keep alive whatever old clients already knew. +There is no required `migrate` ceremony and no required `finalize` — **upgrading +the `secrets` tool *is* the migration.** `finalize` survives only as optional, +indefinitely-deferrable garbage collection. + +Direction chosen over the alternatives: "park it / never finalize" (leaves the +two-format wart in place, resolves nothing) and "abandon the suffix rename" +(reverts shipped EGB-703 behavior, keeps the ugly `gradle-properties` suffix +forever). Additive v2 is the only option that reaches a clean v2 end-state +*without* a coordinated flag-day. + +## Design + +### 1. Read resolution — suffix-agnostic + +Any upgraded client resolving a `properties` external blob looks for the v2 +suffix `.properties.age` first, then falls back to the v1 `.gradle-properties.age`. +An upgraded client therefore **never fails to find a blob** regardless of which +suffix is on disk. Old (pre-0.6.0.0) clients still read v1-only — shipped code +cannot be changed. + +Code: today's `_external_blob_suffix(type)` (a single suffix string) is replaced +by a resolver that returns the **path of the blob that exists** for a given +`(project, slug, type)`, trying v2 then v1. `file` externals are unchanged in +both formats, so the resolver is a no-op identity for them (single suffix +`.file.age`). + +### 2. Write rule — the self-managing twin rule + +On push of a `properties` external, the client checks the store for an existing +v1 twin (`.gradle-properties.age`): + +- **v1 twin exists** (the external was first pushed by an old client) → + **dual-write both suffixes.** Old clients stay fresh forever; nobody is cut off. +- **No v1 twin exists** (brand-new external, first pushed by an upgraded client) → + **write v2-only (`.properties.age`).** Old clients cannot see it → the gentle, + *intended* forcing function. It only ever bites on genuinely new externals, + never on anything that previously worked. + +No stored state is required: the presence/absence of the v1 twin **is** the +signal. Mental model: *"keep alive what old clients already knew; new things are +v2-only."* + +### 3. The marker — auto-stamped, informational only + +The first push by an upgraded client stamps `.secrets-format = 2`. Because writes +are now driven by the twin rule (not the marker), the marker no longer decides +blob location — it becomes purely "a v2-aware client has touched this store." +`secrets which` continues to report `format: vN`. + +### 4. `finalize` → optional GC, never required + +`secrets migrate --finalize` remains the *only* operation that stops dual-writing +and drops the v1 twins. It is pure space reclamation (kilobytes), still +coordination-gated **if** you choose to run it, but never obligatory and +deferrable forever. Its existing safety posture is unchanged (recovery tag, +`verify --all` green gate, twin-before-drop, `--yes`/operator confirmation). This +is what defuses the gate: the destructive step still exists but is now optional +housekeeping, not a release blocker. + +### 5. `migrate` / `migrate --status` → normalize + inspect + +- `secrets migrate` (the EGB-710 manifest-free copy-forward) stays as an optional + "backfill v2 twins for existing v1-only externals" command — useful right + before a `finalize`. With read-fallback (§1) it is no longer required for + correctness, only for tidiness. +- `secrets migrate --status` becomes the **coverage survey**: per external, which + suffixes exist, and whether old clients are still being served (i.e. whether a + v1 twin is still present and being dual-written). This is the operator's + dashboard for "is anyone still relying on v1?" before an optional `finalize`. + +### 6. Propagation semantics (→ user docs verbatim) + +"Has not migrated" means **old client** (`secrets` < 0.6.0.0), not "hasn't run +`migrate`". `migrate` is a per-store op done once by anyone; what protects a +teammate is their **client version**. A read-only teammate needs only the tool +`git pull` (binary ≥ 0.6.0.0), not to run `migrate` themselves. "Upgrade your +secrets" (git pull the tool and/or migrate the store) is the correct umbrella — +and for a solo dev the two are one motion. + +Does a teammate on an old client get a secret User 1 just added? + +| Secret type | Blob name v1 vs v2 | Old client gets the new secret? | +|---|---|---| +| **dotenv** (`.env`, `.env.*`, `.dev.vars`) | identical (`.env.age`) | **Yes, always.** No forcing function possible — the blob name never changed. | +| **`file` external** (keystore, etc.) | identical (`.file.age`) | **Yes, always.** | +| **`properties` external** — existing (has a v1 twin) | dual-written | **Yes.** Old client reads the maintained `.gradle-properties.age`. | +| **`properties` external** — brand-new (no v1 twin) | v2-only | **No → must upgrade.** The forcing function; rare, and never breaks anything that previously worked. | + +Net: a teammate on an old client keeps getting **all** everyday `.env` updates +indefinitely (a good safety property — no one silently misses everyday secrets), +and only hits a wall on a genuinely new `properties`-style external. + +### 7. Known caveat (documented, not engineered around) + +Dual-write keeps v1 *readers* fresh, but a v1 *writer* writes only +`.gradle-properties.age`. A v2 reader (reading `.properties.age` first) could +therefore read stale data until that external is re-pushed by an upgraded client. +For the normal shape — one writer per external, who upgrades first — it never +bites. This matches today's reality and is documented rather than solved +(solving it would require timestamp/newest-wins arbitration across two encrypted +blobs — YAGNI for v1). + +### 8. Backward-compatibility matrix + +| Actor | dotenv / `file` | `properties` (existing twin) | `properties` (new, v2-only) | +|---|---|---|---| +| Upgraded client reads | ✓ | ✓ (resolver finds either) | ✓ | +| Upgraded client writes | unchanged | dual-writes both | writes v2-only | +| Old client reads | ✓ | ✓ (reads maintained v1 twin) | ✗ (forcing function) | +| Old client writes | unchanged | writes v1 twin only (see §7) | n/a (can't create v2) | + +## Code-level surface (for the implementation plan) + +- **`_external_blob_suffix(type)`** → split into: + - `_resolve_external_blob_read(project, slug, type)` — returns the path of the + blob that exists, trying `.properties.age` then `.gradle-properties.age` for + `properties`; identity for `file`. Used by `pull_external_files`, `verify`, + and any read path. + - `_external_blob_write_targets(project, slug, type)` — returns the suffix + path(s) to write: for `properties`, both suffixes when a v1 twin already + exists, else v2-only; single path for `file`. +- **`push_external_files`** — write to every path from + `_external_blob_write_targets` (was a single `age -o`); auto-stamp the marker + on first push by an upgraded client. +- **`pull_external_files` / `cmd_verify`** — resolve blobs via + `_resolve_external_blob_read` (was the single-suffix lookup). +- **`_migrate_finalize`** — unchanged logic; doc/help reframed as optional GC. +- **`_migrate_project` / `_migrate_status`** — retained (EGB-710); `--status` + extended to report dual-write coverage per external. +- **Docs:** CLAUDE.md "Store format" bullet, README, `cmd_help`, CHANGELOG, + VERSION bump (minor — new write semantics). + +## Safety / constraints (unchanged invariants) + +- bash 3.2 portable; every store walk stays recursive (`find -type f`). +- Path-validation rails (`_validate_external_target_path`, slug derivation, + symlink/`..` refusal) untouched. +- `finalize`'s destructive gating (recovery tag, verify-green, twin-before-drop, + `--yes`) untouched. +- The store still carries no manifest; `.secrets.json` remains per-project. + +## Test plan (bats, outline) + +1. Read-fallback: a v2 client resolves a `properties` external that exists only + as `.gradle-properties.age` (no twin) — pull succeeds. +2. Twin rule — existing twin → dual-write: push an external that has a v1 twin; + assert **both** suffixes are written and an old-client read path (v1 suffix) + sees the fresh value. +3. Twin rule — new external → v2-only: push a brand-new `properties` external; + assert **only** `.properties.age` is written (no v1 twin created) — the + forcing function. +4. Marker auto-stamp: first push by an upgraded client on a markerless store + stamps `.secrets-format = 2`. +5. dotenv/`file` unaffected: a dotenv and a `file` external round-trip identically + regardless of store marker. +6. `migrate --status` coverage: reports which externals are dual-written vs + v2-only. +7. `finalize` still green: existing finalize gates and drop behavior unchanged. +8. Caveat is observable (optional): a v1-suffix-only update is read by the v2 + client via fallback (documents the one-writer assumption). + +## Ripple to the roadmap + +- **EGB-703**: "finalize is the destructive milestone you must reach" → "finalize + is optional GC." Update the ticket/notes. +- **EGB-709** (collapse v1 paths): no longer gates on "all clients v2." Becomes + "delete the dual-write/transitional code **if/when** every store is GC'd to + pure v2" — much later, low stakes. Add a note to EGB-709. +- A new ticket should track this work (additive-v2 dual-write). + +## Decided knobs (no longer open) + +- Migration ceremony: **automatic** (upgraded client dual-writes on push; no + required `migrate`). `migrate`/`--status` remain manual/inspection tools. +- Forcing function: **kept**, scoped to brand-new `properties` externals via the + twin rule (existing twins always dual-written). +- `finalize`: **kept** as optional GC (not removed), so a fully-upgraded store + can still be reclaimed to pure v2.