secrets/docs/superpowers/specs/2026-06-08-additive-v2-dual-write-design.md

Additive v2 — automatic dual-write (no flag-day finalize)

Status: Design approved (2026-06-08), pending implementation plan. Supersedes the framing of: EGB-703 ("finalize is the destructive milestone you must reach") and EGB-709 ("delete v1 paths once every client is v2"). Relates to: EGB-677 (storage-model unification), EGB-710 (manifest-free migrate).

Problem

Store-format v2 (EGB-703) renamed the properties external blob suffix (.gradle-properties.age → .properties.age) and made the transition a three-step migration ending in a destructive secrets migrate --finalize that drops the v1 blobs. Finalize is gated on "every machine and store in use is v2" — a coordination requirement that's trivial for a solo dev but uncertain for a distributed team of 2–3+: there is no reliable "all-clear" signal, and finalizing early silently cuts an un-upgraded client off from properties externals (it looks for .gradle-properties.age, which is gone).

The hard gate exists only because finalize permanently drops the v1 blobs. Remove the obligation to drop, and there is nothing to coordinate.

Decision

Adopt Strategy B — Additive v2: "becoming v2" stops being a destructive milestone and becomes a property an upgraded client maintains automatically. Reads try both suffixes; writes keep alive whatever old clients already knew. There is no required migrate ceremony and no required finalize — upgrading the secrets tool is the migration. finalize survives only as optional, indefinitely-deferrable garbage collection.

Direction chosen over the alternatives: "park it / never finalize" (leaves the two-format wart in place, resolves nothing) and "abandon the suffix rename" (reverts shipped EGB-703 behavior, keeps the ugly gradle-properties suffix forever). Additive v2 is the only option that reaches a clean v2 end-state without a coordinated flag-day.

Design

1. Read resolution — suffix-agnostic

Any upgraded client resolving a properties external blob looks for the v2 suffix .properties.age first, then falls back to the v1 .gradle-properties.age. An upgraded client therefore never fails to find a blob regardless of which suffix is on disk. Old (pre-0.6.0.0) clients still read v1-only — shipped code cannot be changed.

Code: today's _external_blob_suffix(type) (a single suffix string) is replaced by a resolver that returns the path of the blob that exists for a given (project, slug, type), trying v2 then v1. file externals are unchanged in both formats, so the resolver is a no-op identity for them (single suffix .file.age).

2. Write rule — the self-managing twin rule

On push of a properties external, the client checks the store for an existing v1 twin (<slug>.gradle-properties.age):

• v1 twin exists (the external was first pushed by an old client) → dual-write both suffixes. Old clients stay fresh forever; nobody is cut off.
• No v1 twin exists (brand-new external, first pushed by an upgraded client) → write v2-only (.properties.age). Old clients cannot see it → the gentle, intended forcing function. It only ever bites on genuinely new externals, never on anything that previously worked.

No stored state is required: the presence/absence of the v1 twin is the signal. Mental model: "keep alive what old clients already knew; new things are v2-only."

3. The marker — auto-stamped, informational only

The first push by an upgraded client stamps .secrets-format = 2. Because writes are now driven by the twin rule (not the marker), the marker no longer decides blob location — it becomes purely "a v2-aware client has touched this store." secrets which continues to report format: vN.

4. finalize → optional GC, never required

secrets migrate --finalize remains the only operation that stops dual-writing and drops the v1 twins. It is pure space reclamation (kilobytes), still coordination-gated if you choose to run it, but never obligatory and deferrable forever. Its existing safety posture is unchanged (recovery tag, verify --all green gate, twin-before-drop, --yes/operator confirmation). This is what defuses the gate: the destructive step still exists but is now optional housekeeping, not a release blocker.

5. migrate / migrate --status → normalize + inspect

• secrets migrate (the EGB-710 manifest-free copy-forward) stays as an optional "backfill v2 twins for existing v1-only externals" command — useful right before a finalize. With read-fallback (§1) it is no longer required for correctness, only for tidiness.
• secrets migrate --status becomes the coverage survey: per external, which suffixes exist, and whether old clients are still being served (i.e. whether a v1 twin is still present and being dual-written). This is the operator's dashboard for "is anyone still relying on v1?" before an optional finalize.

6. Propagation semantics (→ user docs verbatim)

"Has not migrated" means old client (secrets < 0.6.0.0), not "hasn't run migrate". migrate is a per-store op done once by anyone; what protects a teammate is their client version. A read-only teammate needs only the tool git pull (binary ≥ 0.6.0.0), not to run migrate themselves. "Upgrade your secrets" (git pull the tool and/or migrate the store) is the correct umbrella — and for a solo dev the two are one motion.

Does a teammate on an old client get a secret User 1 just added?

Secret type	Blob name v1 vs v2	Old client gets the new secret?
dotenv (.env, .env.*, .dev.vars)	identical (.env.age)	Yes, always. No forcing function possible — the blob name never changed.
file external (keystore, etc.)	identical (.file.age)	Yes, always.
properties external — existing (has a v1 twin)	dual-written	Yes. Old client reads the maintained .gradle-properties.age.
properties external — brand-new (no v1 twin)	v2-only	No → must upgrade. The forcing function; rare, and never breaks anything that previously worked.

Net: a teammate on an old client keeps getting all everyday .env updates indefinitely (a good safety property — no one silently misses everyday secrets), and only hits a wall on a genuinely new properties-style external.

7. Known caveat (documented, not engineered around)

Dual-write keeps v1 readers fresh, but a v1 writer writes only .gradle-properties.age. A v2 reader (reading .properties.age first) could therefore read stale data until that external is re-pushed by an upgraded client. For the normal shape — one writer per external, who upgrades first — it never bites. This matches today's reality and is documented rather than solved (solving it would require timestamp/newest-wins arbitration across two encrypted blobs — YAGNI for v1).

8. Backward-compatibility matrix

Actor	dotenv / file	properties (existing twin)	properties (new, v2-only)
Upgraded client reads	✓	✓ (resolver finds either)	✓
Upgraded client writes	unchanged	dual-writes both	writes v2-only
Old client reads	✓	✓ (reads maintained v1 twin)	✗ (forcing function)
Old client writes	unchanged	writes v1 twin only (see §7)	n/a (can't create v2)

Code-level surface (for the implementation plan)

• _external_blob_suffix(type) → split into:
  • _resolve_external_blob_read(project, slug, type) — returns the path of the blob that exists, trying .properties.age then .gradle-properties.age for properties; identity for file. Used by pull_external_files, verify, and any read path.
  • _external_blob_write_targets(project, slug, type) — returns the suffix path(s) to write: for properties, both suffixes when a v1 twin already exists, else v2-only; single path for file.
• push_external_files — write to every path from _external_blob_write_targets (was a single age -o); auto-stamp the marker on first push by an upgraded client.
• pull_external_files / cmd_verify — resolve blobs via _resolve_external_blob_read (was the single-suffix lookup).
• _migrate_finalize — unchanged logic; doc/help reframed as optional GC.
• _migrate_project / _migrate_status — retained (EGB-710); --status extended to report dual-write coverage per external.
• Docs: CLAUDE.md "Store format" bullet, README, cmd_help, CHANGELOG, VERSION bump (minor — new write semantics).

Safety / constraints (unchanged invariants)

• bash 3.2 portable; every store walk stays recursive (find -type f).
• Path-validation rails (_validate_external_target_path, slug derivation, symlink/.. refusal) untouched.
• finalize's destructive gating (recovery tag, verify-green, twin-before-drop, --yes) untouched.
• The store still carries no manifest; .secrets.json remains per-project.

Test plan (bats, outline)

1. Read-fallback: a v2 client resolves a properties external that exists only as .gradle-properties.age (no twin) — pull succeeds.
2. Twin rule — existing twin → dual-write: push an external that has a v1 twin; assert both suffixes are written and an old-client read path (v1 suffix) sees the fresh value.
3. Twin rule — new external → v2-only: push a brand-new properties external; assert only .properties.age is written (no v1 twin created) — the forcing function.
4. Marker auto-stamp: first push by an upgraded client on a markerless store stamps .secrets-format = 2.
5. dotenv/file unaffected: a dotenv and a file external round-trip identically regardless of store marker.
6. migrate --status coverage: reports which externals are dual-written vs v2-only.
7. finalize still green: existing finalize gates and drop behavior unchanged.
8. Caveat is observable (optional): a v1-suffix-only update is read by the v2 client via fallback (documents the one-writer assumption).

Ripple to the roadmap

• EGB-703: "finalize is the destructive milestone you must reach" → "finalize is optional GC." Update the ticket/notes.
• EGB-709 (collapse v1 paths): no longer gates on "all clients v2." Becomes "delete the dual-write/transitional code if/when every store is GC'd to pure v2" — much later, low stakes. Add a note to EGB-709.
• A new ticket should track this work (additive-v2 dual-write).

Decided knobs (no longer open)

• Migration ceremony: automatic (upgraded client dual-writes on push; no required migrate). migrate/--status remain manual/inspection tools.
• Forcing function: kept, scoped to brand-new properties externals via the twin rule (existing twins always dual-written).
• finalize: kept as optional GC (not removed), so a fully-upgraded store can still be reclaimed to pure v2.