secrets/docs/superpowers/specs/2026-06-08-additive-v2-dual-write-design.md

# Additive v2 — automatic dual-write (no flag-day finalize)

**Status:** Design approved (2026-06-08), pending implementation plan.
**Supersedes the framing of:** EGB-703 ("finalize is the destructive milestone you must reach") and EGB-709 ("delete v1 paths once every client is v2").
**Relates to:** EGB-677 (storage-model unification), EGB-710 (manifest-free migrate).

## Problem

Store-format v2 (EGB-703) renamed the `properties` external blob suffix
(`.gradle-properties.age` → `.properties.age`) and made the transition a
three-step migration ending in a **destructive** `secrets migrate --finalize`
that drops the v1 blobs. Finalize is gated on "every machine and store in use is
v2" — a coordination requirement that's trivial for a solo dev but **uncertain
for a distributed team of 2–3+**: there is no reliable "all-clear" signal, and
finalizing early silently cuts an un-upgraded client off from `properties`
externals (it looks for `.gradle-properties.age`, which is gone).

The hard gate exists **only** because finalize permanently drops the v1 blobs.
Remove the obligation to drop, and there is nothing to coordinate.

## Decision

Adopt **Strategy B — Additive v2**: "becoming v2" stops being a destructive
milestone and becomes a property an upgraded client maintains **automatically**.
Reads try both suffixes; writes keep alive whatever old clients already knew.
There is no required `migrate` ceremony and no required `finalize` — **upgrading
the `secrets` tool *is* the migration.** `finalize` survives only as optional,
indefinitely-deferrable garbage collection.

Direction chosen over the alternatives: "park it / never finalize" (leaves the
two-format wart in place, resolves nothing) and "abandon the suffix rename"
(reverts shipped EGB-703 behavior, keeps the ugly `gradle-properties` suffix
forever). Additive v2 is the only option that reaches a clean v2 end-state
*without* a coordinated flag-day.

## Design

### 1. Read resolution — suffix-agnostic

Any upgraded client resolving a `properties` external blob looks for the v2
suffix `.properties.age` first, then falls back to the v1 `.gradle-properties.age`.
An upgraded client therefore **never fails to find a blob** regardless of which
suffix is on disk. Old (pre-0.6.0.0) clients still read v1-only — shipped code
cannot be changed.

Code: today's `_external_blob_suffix(type)` (a single suffix string) is replaced
by a resolver that returns the **path of the blob that exists** for a given
`(project, slug, type)`, trying v2 then v1. `file` externals are unchanged in
both formats, so the resolver is a no-op identity for them (single suffix
`.file.age`).

### 2. Write rule — the self-managing twin rule

On push of a `properties` external, the client checks the store for an existing
v1 twin (`<slug>.gradle-properties.age`):

- **v1 twin exists** (the external was first pushed by an old client) →
  **dual-write both suffixes.** Old clients stay fresh forever; nobody is cut off.
- **No v1 twin exists** (brand-new external, first pushed by an upgraded client) →
  **write v2-only (`.properties.age`).** Old clients cannot see it → the gentle,
  *intended* forcing function. It only ever bites on genuinely new externals,
  never on anything that previously worked.

No stored state is required: the presence/absence of the v1 twin **is** the
signal. Mental model: *"keep alive what old clients already knew; new things are
v2-only."*

### 3. The marker — unchanged meaning, NOT auto-stamped

Additive-v2 read/write behavior does **not** depend on the `.secrets-format`
marker at all (reads try both suffixes; writes follow the twin rule). So the
marker is left exactly as it is today: set only by `init` (born-v2, a fresh pure-v2
store) or `migrate --finalize` (a store GC'd to pure v2). A transitional store
that upgraded clients are dual-writing stays **markerless (v1)** — which is
*accurate*: it has not been finalized to pure v2, and v1 twins still exist.

Push deliberately does **not** auto-stamp the marker. (An earlier draft proposed
auto-stamping on first push; that was dropped during planning because it would
make a v1 store read as v2 the moment anyone pushed — turning every `migrate`
into a no-op and contradicting the "v1 until finalized" model the whole
migration relies on. The marker's only purposes — `secrets which` display and
gating `migrate`/`finalize` — are better served by it continuing to mean
"finalized/pure v2.")

`secrets which` continues to report `format: vN` from the marker.

### 4. `finalize` → optional GC, never required

`secrets migrate --finalize` remains the *only* operation that stops dual-writing
and drops the v1 twins. It is pure space reclamation (kilobytes), still
coordination-gated **if** you choose to run it, but never obligatory and
deferrable forever. Its existing safety posture is unchanged (recovery tag,
`verify --all` green gate, twin-before-drop, `--yes`/operator confirmation). This
is what defuses the gate: the destructive step still exists but is now optional
housekeeping, not a release blocker.

### 5. `migrate` / `migrate --status` → normalize + inspect

- `secrets migrate` (the EGB-710 manifest-free copy-forward) stays as an optional
  "backfill v2 twins for existing v1-only externals" command — useful right
  before a `finalize`. With read-fallback (§1) it is no longer required for
  correctness, only for tidiness.
- `secrets migrate --status` becomes the **coverage survey**: per external, which
  suffixes exist, and whether old clients are still being served (i.e. whether a
  v1 twin is still present and being dual-written). This is the operator's
  dashboard for "is anyone still relying on v1?" before an optional `finalize`.

### 6. Propagation semantics (→ user docs verbatim)

"Has not migrated" means **old client** (`secrets` < 0.6.0.0), not "hasn't run
`migrate`". `migrate` is a per-store op done once by anyone; what protects a
teammate is their **client version**. A read-only teammate needs only the tool
`git pull` (binary ≥ 0.6.0.0), not to run `migrate` themselves. "Upgrade your
secrets" (git pull the tool and/or migrate the store) is the correct umbrella —
and for a solo dev the two are one motion.

Does a teammate on an old client get a secret User 1 just added?

| Secret type | Blob name v1 vs v2 | Old client gets the new secret? |
|---|---|---|
| **dotenv** (`.env`, `.env.*`, `.dev.vars`) | identical (`.env.age`) | **Yes, always.** No forcing function possible — the blob name never changed. |
| **`file` external** (keystore, etc.) | identical (`.file.age`) | **Yes, always.** |
| **`properties` external** — existing (has a v1 twin) | dual-written | **Yes.** Old client reads the maintained `.gradle-properties.age`. |
| **`properties` external** — brand-new (no v1 twin) | v2-only | **No → must upgrade.** The forcing function; rare, and never breaks anything that previously worked. |

Net: a teammate on an old client keeps getting **all** everyday `.env` updates
indefinitely (a good safety property — no one silently misses everyday secrets),
and only hits a wall on a genuinely new `properties`-style external.

### 7. Known caveat (documented, not engineered around)

Dual-write keeps v1 *readers* fresh, but a v1 *writer* writes only
`.gradle-properties.age`. A v2 reader (reading `.properties.age` first) could
therefore read stale data until that external is re-pushed by an upgraded client.
For the normal shape — one writer per external, who upgrades first — it never
bites. This matches today's reality and is documented rather than solved
(solving it would require timestamp/newest-wins arbitration across two encrypted
blobs — YAGNI for v1).

### 8. Backward-compatibility matrix

| Actor | dotenv / `file` | `properties` (existing twin) | `properties` (new, v2-only) |
|---|---|---|---|
| Upgraded client reads | ✓ | ✓ (resolver finds either) | ✓ |
| Upgraded client writes | unchanged | dual-writes both | writes v2-only |
| Old client reads | ✓ | ✓ (reads maintained v1 twin) | ✗ (forcing function) |
| Old client writes | unchanged | writes v1 twin only (see §7) | n/a (can't create v2) |

## Code-level surface (for the implementation plan)

- **`_external_blob_suffix(type)`** → split into:
  - `_resolve_external_blob_read(project, slug, type)` — returns the path of the
    blob that exists, trying `.properties.age` then `.gradle-properties.age` for
    `properties`; identity for `file`. Used by `pull_external_files`, `verify`,
    and any read path.
  - `_external_blob_write_targets(project, slug, type)` — returns the suffix
    path(s) to write: for `properties`, both suffixes when a v1 twin already
    exists, else v2-only; single path for `file`.
- **`push_external_files`** — write to every path from
  `_external_blob_write_targets` (was a single `age -o`). Push does **not** stamp
  the marker (see §3).
- **`pull_external_files` / `cmd_verify`** — resolve blobs via
  `_resolve_external_blob_read` (was the single-suffix lookup).
- **`_migrate_finalize`** — unchanged logic; doc/help reframed as optional GC.
- **`_migrate_project` / `_migrate_status`** — retained (EGB-710); `--status`
  extended to report dual-write coverage per external.
- **Docs:** CLAUDE.md "Store format" bullet, README, `cmd_help`, CHANGELOG,
  VERSION bump (minor — new write semantics).

## Safety / constraints (unchanged invariants)

- bash 3.2 portable; every store walk stays recursive (`find -type f`).
- Path-validation rails (`_validate_external_target_path`, slug derivation,
  symlink/`..` refusal) untouched.
- `finalize`'s destructive gating (recovery tag, verify-green, twin-before-drop,
  `--yes`) untouched.
- The store still carries no manifest; `.secrets.json` remains per-project.

## Test plan (bats, outline)

1. Read-fallback: a v2 client resolves a `properties` external that exists only
   as `.gradle-properties.age` (no twin) — pull succeeds.
2. Twin rule — existing twin → dual-write: push an external that has a v1 twin;
   assert **both** suffixes are written and an old-client read path (v1 suffix)
   sees the fresh value.
3. Twin rule — new external → v2-only: push a brand-new `properties` external;
   assert **only** `.properties.age` is written (no v1 twin created) — the
   forcing function.
4. Marker NOT stamped on push: a `make_v1_store` + push leaves `.secrets-format`
   absent (store stays v1/transitional); the existing `make_v1_store; push;
   migrate` flow is unaffected.
5. dotenv/`file` unaffected: a dotenv and a `file` external round-trip identically
   regardless of store marker.
6. `migrate --status` coverage: reports which externals are dual-written vs
   v2-only.
7. `finalize` still green: existing finalize gates and drop behavior unchanged.
8. Caveat is observable (optional): a v1-suffix-only update is read by the v2
   client via fallback (documents the one-writer assumption).

## Ripple to the roadmap

- **EGB-703**: "finalize is the destructive milestone you must reach" → "finalize
  is optional GC." Update the ticket/notes.
- **EGB-709** (collapse v1 paths): no longer gates on "all clients v2." Becomes
  "delete the dual-write/transitional code **if/when** every store is GC'd to
  pure v2" — much later, low stakes. Add a note to EGB-709.
- A new ticket should track this work (additive-v2 dual-write).

## Decided knobs (no longer open)

- Migration ceremony: **automatic** (upgraded client dual-writes on push; no
  required `migrate`). `migrate`/`--status` remain manual/inspection tools.
- Forcing function: **kept**, scoped to brand-new `properties` externals via the
  twin rule (existing twins always dual-written).
- `finalize`: **kept** as optional GC (not removed), so a fully-upgraded store
  can still be reclaimed to pure v2.