docs: additive-v2 dual-write design spec (defuse the finalize gate)
This commit is contained in:
Brian Majewski
parent
836b418a12
commit
54575af61c
1 changed files with 203 additions and 0 deletions
|
|
@ -0,0 +1,203 @@
|
||||||
|
# Additive v2 — automatic dual-write (no flag-day finalize)
|
||||||
|
|
||||||
|
**Status:** Design approved (2026-06-08), pending implementation plan.
|
||||||
|
**Supersedes the framing of:** EGB-703 ("finalize is the destructive milestone you must reach") and EGB-709 ("delete v1 paths once every client is v2").
|
||||||
|
**Relates to:** EGB-677 (storage-model unification), EGB-710 (manifest-free migrate).
|
||||||
|
|
||||||
|
## Problem
|
||||||
|
|
||||||
|
Store-format v2 (EGB-703) renamed the `properties` external blob suffix
|
||||||
|
(`.gradle-properties.age` → `.properties.age`) and made the transition a
|
||||||
|
three-step migration ending in a **destructive** `secrets migrate --finalize`
|
||||||
|
that drops the v1 blobs. Finalize is gated on "every machine and store in use is
|
||||||
|
v2" — a coordination requirement that's trivial for a solo dev but **uncertain
|
||||||
|
for a distributed team of 2–3+**: there is no reliable "all-clear" signal, and
|
||||||
|
finalizing early silently cuts an un-upgraded client off from `properties`
|
||||||
|
externals (it looks for `.gradle-properties.age`, which is gone).
|
||||||
|
|
||||||
|
The hard gate exists **only** because finalize permanently drops the v1 blobs.
|
||||||
|
Remove the obligation to drop, and there is nothing to coordinate.
|
||||||
|
|
||||||
|
## Decision
|
||||||
|
|
||||||
|
Adopt **Strategy B — Additive v2**: "becoming v2" stops being a destructive
|
||||||
|
milestone and becomes a property an upgraded client maintains **automatically**.
|
||||||
|
Reads try both suffixes; writes keep alive whatever old clients already knew.
|
||||||
|
There is no required `migrate` ceremony and no required `finalize` — **upgrading
|
||||||
|
the `secrets` tool *is* the migration.** `finalize` survives only as optional,
|
||||||
|
indefinitely-deferrable garbage collection.
|
||||||
|
|
||||||
|
Direction chosen over the alternatives: "park it / never finalize" (leaves the
|
||||||
|
two-format wart in place, resolves nothing) and "abandon the suffix rename"
|
||||||
|
(reverts shipped EGB-703 behavior, keeps the ugly `gradle-properties` suffix
|
||||||
|
forever). Additive v2 is the only option that reaches a clean v2 end-state
|
||||||
|
*without* a coordinated flag-day.
|
||||||
|
|
||||||
|
## Design
|
||||||
|
|
||||||
|
### 1. Read resolution — suffix-agnostic
|
||||||
|
|
||||||
|
Any upgraded client resolving a `properties` external blob looks for the v2
|
||||||
|
suffix `.properties.age` first, then falls back to the v1 `.gradle-properties.age`.
|
||||||
|
An upgraded client therefore **never fails to find a blob** regardless of which
|
||||||
|
suffix is on disk. Old (pre-0.6.0.0) clients still read v1-only — shipped code
|
||||||
|
cannot be changed.
|
||||||
|
|
||||||
|
Code: today's `_external_blob_suffix(type)` (a single suffix string) is replaced
|
||||||
|
by a resolver that returns the **path of the blob that exists** for a given
|
||||||
|
`(project, slug, type)`, trying v2 then v1. `file` externals are unchanged in
|
||||||
|
both formats, so the resolver is a no-op identity for them (single suffix
|
||||||
|
`.file.age`).
|
||||||
|
|
||||||
|
### 2. Write rule — the self-managing twin rule
|
||||||
|
|
||||||
|
On push of a `properties` external, the client checks the store for an existing
|
||||||
|
v1 twin (`<slug>.gradle-properties.age`):
|
||||||
|
|
||||||
|
- **v1 twin exists** (the external was first pushed by an old client) →
|
||||||
|
**dual-write both suffixes.** Old clients stay fresh forever; nobody is cut off.
|
||||||
|
- **No v1 twin exists** (brand-new external, first pushed by an upgraded client) →
|
||||||
|
**write v2-only (`.properties.age`).** Old clients cannot see it → the gentle,
|
||||||
|
*intended* forcing function. It only ever bites on genuinely new externals,
|
||||||
|
never on anything that previously worked.
|
||||||
|
|
||||||
|
No stored state is required: the presence/absence of the v1 twin **is** the
|
||||||
|
signal. Mental model: *"keep alive what old clients already knew; new things are
|
||||||
|
v2-only."*
|
||||||
|
|
||||||
|
### 3. The marker — auto-stamped, informational only
|
||||||
|
|
||||||
|
The first push by an upgraded client stamps `.secrets-format = 2`. Because writes
|
||||||
|
are now driven by the twin rule (not the marker), the marker no longer decides
|
||||||
|
blob location — it becomes purely "a v2-aware client has touched this store."
|
||||||
|
`secrets which` continues to report `format: vN`.
|
||||||
|
|
||||||
|
### 4. `finalize` → optional GC, never required
|
||||||
|
|
||||||
|
`secrets migrate --finalize` remains the *only* operation that stops dual-writing
|
||||||
|
and drops the v1 twins. It is pure space reclamation (kilobytes), still
|
||||||
|
coordination-gated **if** you choose to run it, but never obligatory and
|
||||||
|
deferrable forever. Its existing safety posture is unchanged (recovery tag,
|
||||||
|
`verify --all` green gate, twin-before-drop, `--yes`/operator confirmation). This
|
||||||
|
is what defuses the gate: the destructive step still exists but is now optional
|
||||||
|
housekeeping, not a release blocker.
|
||||||
|
|
||||||
|
### 5. `migrate` / `migrate --status` → normalize + inspect
|
||||||
|
|
||||||
|
- `secrets migrate` (the EGB-710 manifest-free copy-forward) stays as an optional
|
||||||
|
"backfill v2 twins for existing v1-only externals" command — useful right
|
||||||
|
before a `finalize`. With read-fallback (§1) it is no longer required for
|
||||||
|
correctness, only for tidiness.
|
||||||
|
- `secrets migrate --status` becomes the **coverage survey**: per external, which
|
||||||
|
suffixes exist, and whether old clients are still being served (i.e. whether a
|
||||||
|
v1 twin is still present and being dual-written). This is the operator's
|
||||||
|
dashboard for "is anyone still relying on v1?" before an optional `finalize`.
|
||||||
|
|
||||||
|
### 6. Propagation semantics (→ user docs verbatim)
|
||||||
|
|
||||||
|
"Has not migrated" means **old client** (`secrets` < 0.6.0.0), not "hasn't run
|
||||||
|
`migrate`". `migrate` is a per-store op done once by anyone; what protects a
|
||||||
|
teammate is their **client version**. A read-only teammate needs only the tool
|
||||||
|
`git pull` (binary ≥ 0.6.0.0), not to run `migrate` themselves. "Upgrade your
|
||||||
|
secrets" (git pull the tool and/or migrate the store) is the correct umbrella —
|
||||||
|
and for a solo dev the two are one motion.
|
||||||
|
|
||||||
|
Does a teammate on an old client get a secret User 1 just added?
|
||||||
|
|
||||||
|
| Secret type | Blob name v1 vs v2 | Old client gets the new secret? |
|
||||||
|
|---|---|---|
|
||||||
|
| **dotenv** (`.env`, `.env.*`, `.dev.vars`) | identical (`.env.age`) | **Yes, always.** No forcing function possible — the blob name never changed. |
|
||||||
|
| **`file` external** (keystore, etc.) | identical (`.file.age`) | **Yes, always.** |
|
||||||
|
| **`properties` external** — existing (has a v1 twin) | dual-written | **Yes.** Old client reads the maintained `.gradle-properties.age`. |
|
||||||
|
| **`properties` external** — brand-new (no v1 twin) | v2-only | **No → must upgrade.** The forcing function; rare, and never breaks anything that previously worked. |
|
||||||
|
|
||||||
|
Net: a teammate on an old client keeps getting **all** everyday `.env` updates
|
||||||
|
indefinitely (a good safety property — no one silently misses everyday secrets),
|
||||||
|
and only hits a wall on a genuinely new `properties`-style external.
|
||||||
|
|
||||||
|
### 7. Known caveat (documented, not engineered around)
|
||||||
|
|
||||||
|
Dual-write keeps v1 *readers* fresh, but a v1 *writer* writes only
|
||||||
|
`.gradle-properties.age`. A v2 reader (reading `.properties.age` first) could
|
||||||
|
therefore read stale data until that external is re-pushed by an upgraded client.
|
||||||
|
For the normal shape — one writer per external, who upgrades first — it never
|
||||||
|
bites. This matches today's reality and is documented rather than solved
|
||||||
|
(solving it would require timestamp/newest-wins arbitration across two encrypted
|
||||||
|
blobs — YAGNI for v1).
|
||||||
|
|
||||||
|
### 8. Backward-compatibility matrix
|
||||||
|
|
||||||
|
| Actor | dotenv / `file` | `properties` (existing twin) | `properties` (new, v2-only) |
|
||||||
|
|---|---|---|---|
|
||||||
|
| Upgraded client reads | ✓ | ✓ (resolver finds either) | ✓ |
|
||||||
|
| Upgraded client writes | unchanged | dual-writes both | writes v2-only |
|
||||||
|
| Old client reads | ✓ | ✓ (reads maintained v1 twin) | ✗ (forcing function) |
|
||||||
|
| Old client writes | unchanged | writes v1 twin only (see §7) | n/a (can't create v2) |
|
||||||
|
|
||||||
|
## Code-level surface (for the implementation plan)
|
||||||
|
|
||||||
|
- **`_external_blob_suffix(type)`** → split into:
|
||||||
|
- `_resolve_external_blob_read(project, slug, type)` — returns the path of the
|
||||||
|
blob that exists, trying `.properties.age` then `.gradle-properties.age` for
|
||||||
|
`properties`; identity for `file`. Used by `pull_external_files`, `verify`,
|
||||||
|
and any read path.
|
||||||
|
- `_external_blob_write_targets(project, slug, type)` — returns the suffix
|
||||||
|
path(s) to write: for `properties`, both suffixes when a v1 twin already
|
||||||
|
exists, else v2-only; single path for `file`.
|
||||||
|
- **`push_external_files`** — write to every path from
|
||||||
|
`_external_blob_write_targets` (was a single `age -o`); auto-stamp the marker
|
||||||
|
on first push by an upgraded client.
|
||||||
|
- **`pull_external_files` / `cmd_verify`** — resolve blobs via
|
||||||
|
`_resolve_external_blob_read` (was the single-suffix lookup).
|
||||||
|
- **`_migrate_finalize`** — unchanged logic; doc/help reframed as optional GC.
|
||||||
|
- **`_migrate_project` / `_migrate_status`** — retained (EGB-710); `--status`
|
||||||
|
extended to report dual-write coverage per external.
|
||||||
|
- **Docs:** CLAUDE.md "Store format" bullet, README, `cmd_help`, CHANGELOG,
|
||||||
|
VERSION bump (minor — new write semantics).
|
||||||
|
|
||||||
|
## Safety / constraints (unchanged invariants)
|
||||||
|
|
||||||
|
- bash 3.2 portable; every store walk stays recursive (`find -type f`).
|
||||||
|
- Path-validation rails (`_validate_external_target_path`, slug derivation,
|
||||||
|
symlink/`..` refusal) untouched.
|
||||||
|
- `finalize`'s destructive gating (recovery tag, verify-green, twin-before-drop,
|
||||||
|
`--yes`) untouched.
|
||||||
|
- The store still carries no manifest; `.secrets.json` remains per-project.
|
||||||
|
|
||||||
|
## Test plan (bats, outline)
|
||||||
|
|
||||||
|
1. Read-fallback: a v2 client resolves a `properties` external that exists only
|
||||||
|
as `.gradle-properties.age` (no twin) — pull succeeds.
|
||||||
|
2. Twin rule — existing twin → dual-write: push an external that has a v1 twin;
|
||||||
|
assert **both** suffixes are written and an old-client read path (v1 suffix)
|
||||||
|
sees the fresh value.
|
||||||
|
3. Twin rule — new external → v2-only: push a brand-new `properties` external;
|
||||||
|
assert **only** `.properties.age` is written (no v1 twin created) — the
|
||||||
|
forcing function.
|
||||||
|
4. Marker auto-stamp: first push by an upgraded client on a markerless store
|
||||||
|
stamps `.secrets-format = 2`.
|
||||||
|
5. dotenv/`file` unaffected: a dotenv and a `file` external round-trip identically
|
||||||
|
regardless of store marker.
|
||||||
|
6. `migrate --status` coverage: reports which externals are dual-written vs
|
||||||
|
v2-only.
|
||||||
|
7. `finalize` still green: existing finalize gates and drop behavior unchanged.
|
||||||
|
8. Caveat is observable (optional): a v1-suffix-only update is read by the v2
|
||||||
|
client via fallback (documents the one-writer assumption).
|
||||||
|
|
||||||
|
## Ripple to the roadmap
|
||||||
|
|
||||||
|
- **EGB-703**: "finalize is the destructive milestone you must reach" → "finalize
|
||||||
|
is optional GC." Update the ticket/notes.
|
||||||
|
- **EGB-709** (collapse v1 paths): no longer gates on "all clients v2." Becomes
|
||||||
|
"delete the dual-write/transitional code **if/when** every store is GC'd to
|
||||||
|
pure v2" — much later, low stakes. Add a note to EGB-709.
|
||||||
|
- A new ticket should track this work (additive-v2 dual-write).
|
||||||
|
|
||||||
|
## Decided knobs (no longer open)
|
||||||
|
|
||||||
|
- Migration ceremony: **automatic** (upgraded client dual-writes on push; no
|
||||||
|
required `migrate`). `migrate`/`--status` remain manual/inspection tools.
|
||||||
|
- Forcing function: **kept**, scoped to brand-new `properties` externals via the
|
||||||
|
twin rule (existing twins always dual-written).
|
||||||
|
- `finalize`: **kept** as optional GC (not removed), so a fully-upgraded store
|
||||||
|
can still be reclaimed to pure v2.
|
||||||
Loading…
Add table
Add a link
Reference in a new issue