Merge origin/main into EGB-283 (multi-recipient age encryption)
Reconcile the multi-recipient branch (cut from v0.6.1.0) with origin/main
at v0.7.4.0. The two feature lines are largely orthogonal; the one real
integration point is the external-blob encrypt path:
- EGB-712 added an additive-v2 dual-write loop (_external_blob_write_targets,
writing v2 + any v1 twin). EGB-283 routes every encrypt site through
RECIPIENT_ARGS for N-recipient encryption. Resolution keeps the dual-write
loop but encrypts each target to the full recipient set
(age "${RECIPIENT_ARGS[@]}" per write target), so dual-write and
multi-recipient compose. cmd_push loads recipients before both external
push sites; legacy single-key rekey keeps its fresh-keypair pubkey path.
Version: 0.6.2.0 + 0.7.4.0 -> 0.7.5.0. Docs (CLAUDE.md/README/CHANGELOG)
merged to carry both feature sets; subcommand list now includes
recipients/reencrypt and upgrade.
Tests: full `bats test/` green except 6 pre-existing host-environment
failures (4 chmod-600 restore assertions + 2 jq-PATH-shadow tests, all
macOS-authored), none touching merged code. recipients.bats 34/34 pass;
external/dual-write area passes except the same mode-600 host artifacts.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Brian Majewski
commit
e29024bd63
16 changed files with 2723 additions and 142 deletions
|
|
@ -0,0 +1,216 @@
|
|||
# Additive v2 — automatic dual-write (no flag-day finalize)
|
||||
|
||||
**Status:** Design approved (2026-06-08), pending implementation plan.
|
||||
**Supersedes the framing of:** EGB-703 ("finalize is the destructive milestone you must reach") and EGB-709 ("delete v1 paths once every client is v2").
|
||||
**Relates to:** EGB-677 (storage-model unification), EGB-710 (manifest-free migrate).
|
||||
|
||||
## Problem
|
||||
|
||||
Store-format v2 (EGB-703) renamed the `properties` external blob suffix
|
||||
(`.gradle-properties.age` → `.properties.age`) and made the transition a
|
||||
three-step migration ending in a **destructive** `secrets migrate --finalize`
|
||||
that drops the v1 blobs. Finalize is gated on "every machine and store in use is
|
||||
v2" — a coordination requirement that's trivial for a solo dev but **uncertain
|
||||
for a distributed team of 2–3+**: there is no reliable "all-clear" signal, and
|
||||
finalizing early silently cuts an un-upgraded client off from `properties`
|
||||
externals (it looks for `.gradle-properties.age`, which is gone).
|
||||
|
||||
The hard gate exists **only** because finalize permanently drops the v1 blobs.
|
||||
Remove the obligation to drop, and there is nothing to coordinate.
|
||||
|
||||
## Decision
|
||||
|
||||
Adopt **Strategy B — Additive v2**: "becoming v2" stops being a destructive
|
||||
milestone and becomes a property an upgraded client maintains **automatically**.
|
||||
Reads try both suffixes; writes keep alive whatever old clients already knew.
|
||||
There is no required `migrate` ceremony and no required `finalize` — **upgrading
|
||||
the `secrets` tool *is* the migration.** `finalize` survives only as optional,
|
||||
indefinitely-deferrable garbage collection.
|
||||
|
||||
Direction chosen over the alternatives: "park it / never finalize" (leaves the
|
||||
two-format wart in place, resolves nothing) and "abandon the suffix rename"
|
||||
(reverts shipped EGB-703 behavior, keeps the ugly `gradle-properties` suffix
|
||||
forever). Additive v2 is the only option that reaches a clean v2 end-state
|
||||
*without* a coordinated flag-day.
|
||||
|
||||
## Design
|
||||
|
||||
### 1. Read resolution — suffix-agnostic
|
||||
|
||||
Any upgraded client resolving a `properties` external blob looks for the v2
|
||||
suffix `.properties.age` first, then falls back to the v1 `.gradle-properties.age`.
|
||||
An upgraded client therefore **never fails to find a blob** regardless of which
|
||||
suffix is on disk. Old (pre-0.6.0.0) clients still read v1-only — shipped code
|
||||
cannot be changed.
|
||||
|
||||
Code: today's `_external_blob_suffix(type)` (a single suffix string) is replaced
|
||||
by a resolver that returns the **path of the blob that exists** for a given
|
||||
`(project, slug, type)`, trying v2 then v1. `file` externals are unchanged in
|
||||
both formats, so the resolver is a no-op identity for them (single suffix
|
||||
`.file.age`).
|
||||
|
||||
### 2. Write rule — the self-managing twin rule
|
||||
|
||||
On push of a `properties` external, the client checks the store for an existing
|
||||
v1 twin (`<slug>.gradle-properties.age`):
|
||||
|
||||
- **v1 twin exists** (the external was first pushed by an old client) →
|
||||
**dual-write both suffixes.** Old clients stay fresh forever; nobody is cut off.
|
||||
- **No v1 twin exists** (brand-new external, first pushed by an upgraded client) →
|
||||
**write v2-only (`.properties.age`).** Old clients cannot see it → the gentle,
|
||||
*intended* forcing function. It only ever bites on genuinely new externals,
|
||||
never on anything that previously worked.
|
||||
|
||||
No stored state is required: the presence/absence of the v1 twin **is** the
|
||||
signal. Mental model: *"keep alive what old clients already knew; new things are
|
||||
v2-only."*
|
||||
|
||||
### 3. The marker — unchanged meaning, NOT auto-stamped
|
||||
|
||||
Additive-v2 read/write behavior does **not** depend on the `.secrets-format`
|
||||
marker at all (reads try both suffixes; writes follow the twin rule). So the
|
||||
marker is left exactly as it is today: set only by `init` (born-v2, a fresh pure-v2
|
||||
store) or `migrate --finalize` (a store GC'd to pure v2). A transitional store
|
||||
that upgraded clients are dual-writing stays **markerless (v1)** — which is
|
||||
*accurate*: it has not been finalized to pure v2, and v1 twins still exist.
|
||||
|
||||
Push deliberately does **not** auto-stamp the marker. (An earlier draft proposed
|
||||
auto-stamping on first push; that was dropped during planning because it would
|
||||
make a v1 store read as v2 the moment anyone pushed — turning every `migrate`
|
||||
into a no-op and contradicting the "v1 until finalized" model the whole
|
||||
migration relies on. The marker's only purposes — `secrets which` display and
|
||||
gating `migrate`/`finalize` — are better served by it continuing to mean
|
||||
"finalized/pure v2.")
|
||||
|
||||
`secrets which` continues to report `format: vN` from the marker.
|
||||
|
||||
### 4. `finalize` → optional GC, never required
|
||||
|
||||
`secrets migrate --finalize` remains the *only* operation that stops dual-writing
|
||||
and drops the v1 twins. It is pure space reclamation (kilobytes), still
|
||||
coordination-gated **if** you choose to run it, but never obligatory and
|
||||
deferrable forever. Its existing safety posture is unchanged (recovery tag,
|
||||
`verify --all` green gate, twin-before-drop, `--yes`/operator confirmation). This
|
||||
is what defuses the gate: the destructive step still exists but is now optional
|
||||
housekeeping, not a release blocker.
|
||||
|
||||
### 5. `migrate` / `migrate --status` → normalize + inspect
|
||||
|
||||
- `secrets migrate` (the EGB-710 manifest-free copy-forward) stays as an optional
|
||||
"backfill v2 twins for existing v1-only externals" command — useful right
|
||||
before a `finalize`. With read-fallback (§1) it is no longer required for
|
||||
correctness, only for tidiness.
|
||||
- `secrets migrate --status` becomes the **coverage survey**: per external, which
|
||||
suffixes exist, and whether old clients are still being served (i.e. whether a
|
||||
v1 twin is still present and being dual-written). This is the operator's
|
||||
dashboard for "is anyone still relying on v1?" before an optional `finalize`.
|
||||
|
||||
### 6. Propagation semantics (→ user docs verbatim)
|
||||
|
||||
"Has not migrated" means **old client** (`secrets` < 0.6.0.0), not "hasn't run
|
||||
`migrate`". `migrate` is a per-store op done once by anyone; what protects a
|
||||
teammate is their **client version**. A read-only teammate needs only the tool
|
||||
`git pull` (binary ≥ 0.6.0.0), not to run `migrate` themselves. "Upgrade your
|
||||
secrets" (git pull the tool and/or migrate the store) is the correct umbrella —
|
||||
and for a solo dev the two are one motion.
|
||||
|
||||
Does a teammate on an old client get a secret User 1 just added?
|
||||
|
||||
| Secret type | Blob name v1 vs v2 | Old client gets the new secret? |
|
||||
|---|---|---|
|
||||
| **dotenv** (`.env`, `.env.*`, `.dev.vars`) | identical (`.env.age`) | **Yes, always.** No forcing function possible — the blob name never changed. |
|
||||
| **`file` external** (keystore, etc.) | identical (`.file.age`) | **Yes, always.** |
|
||||
| **`properties` external** — existing (has a v1 twin) | dual-written | **Yes.** Old client reads the maintained `.gradle-properties.age`. |
|
||||
| **`properties` external** — brand-new (no v1 twin) | v2-only | **No → must upgrade.** The forcing function; rare, and never breaks anything that previously worked. |
|
||||
|
||||
Net: a teammate on an old client keeps getting **all** everyday `.env` updates
|
||||
indefinitely (a good safety property — no one silently misses everyday secrets),
|
||||
and only hits a wall on a genuinely new `properties`-style external.
|
||||
|
||||
### 7. Known caveat (documented, not engineered around)
|
||||
|
||||
Dual-write keeps v1 *readers* fresh, but a v1 *writer* writes only
|
||||
`.gradle-properties.age`. A v2 reader (reading `.properties.age` first) could
|
||||
therefore read stale data until that external is re-pushed by an upgraded client.
|
||||
For the normal shape — one writer per external, who upgrades first — it never
|
||||
bites. This matches today's reality and is documented rather than solved
|
||||
(solving it would require timestamp/newest-wins arbitration across two encrypted
|
||||
blobs — YAGNI for v1).
|
||||
|
||||
### 8. Backward-compatibility matrix
|
||||
|
||||
| Actor | dotenv / `file` | `properties` (existing twin) | `properties` (new, v2-only) |
|
||||
|---|---|---|---|
|
||||
| Upgraded client reads | ✓ | ✓ (resolver finds either) | ✓ |
|
||||
| Upgraded client writes | unchanged | dual-writes both | writes v2-only |
|
||||
| Old client reads | ✓ | ✓ (reads maintained v1 twin) | ✗ (forcing function) |
|
||||
| Old client writes | unchanged | writes v1 twin only (see §7) | n/a (can't create v2) |
|
||||
|
||||
## Code-level surface (for the implementation plan)
|
||||
|
||||
- **`_external_blob_suffix(type)`** → split into:
|
||||
- `_resolve_external_blob_read(project, slug, type)` — returns the path of the
|
||||
blob that exists, trying `.properties.age` then `.gradle-properties.age` for
|
||||
`properties`; identity for `file`. Used by `pull_external_files`, `verify`,
|
||||
and any read path.
|
||||
- `_external_blob_write_targets(project, slug, type)` — returns the suffix
|
||||
path(s) to write: for `properties`, both suffixes when a v1 twin already
|
||||
exists, else v2-only; single path for `file`.
|
||||
- **`push_external_files`** — write to every path from
|
||||
`_external_blob_write_targets` (was a single `age -o`). Push does **not** stamp
|
||||
the marker (see §3).
|
||||
- **`pull_external_files` / `cmd_verify`** — resolve blobs via
|
||||
`_resolve_external_blob_read` (was the single-suffix lookup).
|
||||
- **`_migrate_finalize`** — unchanged logic; doc/help reframed as optional GC.
|
||||
- **`_migrate_project` / `_migrate_status`** — retained (EGB-710); `--status`
|
||||
extended to report dual-write coverage per external.
|
||||
- **Docs:** CLAUDE.md "Store format" bullet, README, `cmd_help`, CHANGELOG,
|
||||
VERSION bump (minor — new write semantics).
|
||||
|
||||
## Safety / constraints (unchanged invariants)
|
||||
|
||||
- bash 3.2 portable; every store walk stays recursive (`find -type f`).
|
||||
- Path-validation rails (`_validate_external_target_path`, slug derivation,
|
||||
symlink/`..` refusal) untouched.
|
||||
- `finalize`'s destructive gating (recovery tag, verify-green, twin-before-drop,
|
||||
`--yes`) untouched.
|
||||
- The store still carries no manifest; `.secrets.json` remains per-project.
|
||||
|
||||
## Test plan (bats, outline)
|
||||
|
||||
1. Read-fallback: a v2 client resolves a `properties` external that exists only
|
||||
as `.gradle-properties.age` (no twin) — pull succeeds.
|
||||
2. Twin rule — existing twin → dual-write: push an external that has a v1 twin;
|
||||
assert **both** suffixes are written and an old-client read path (v1 suffix)
|
||||
sees the fresh value.
|
||||
3. Twin rule — new external → v2-only: push a brand-new `properties` external;
|
||||
assert **only** `.properties.age` is written (no v1 twin created) — the
|
||||
forcing function.
|
||||
4. Marker NOT stamped on push: a `make_v1_store` + push leaves `.secrets-format`
|
||||
absent (store stays v1/transitional); the existing `make_v1_store; push;
|
||||
migrate` flow is unaffected.
|
||||
5. dotenv/`file` unaffected: a dotenv and a `file` external round-trip identically
|
||||
regardless of store marker.
|
||||
6. `migrate --status` coverage: reports which externals are dual-written vs
|
||||
v2-only.
|
||||
7. `finalize` still green: existing finalize gates and drop behavior unchanged.
|
||||
8. Caveat is observable (optional): a v1-suffix-only update is read by the v2
|
||||
client via fallback (documents the one-writer assumption).
|
||||
|
||||
## Ripple to the roadmap
|
||||
|
||||
- **EGB-703**: "finalize is the destructive milestone you must reach" → "finalize
|
||||
is optional GC." Update the ticket/notes.
|
||||
- **EGB-709** (collapse v1 paths): no longer gates on "all clients v2." Becomes
|
||||
"delete the dual-write/transitional code **if/when** every store is GC'd to
|
||||
pure v2" — much later, low stakes. Add a note to EGB-709.
|
||||
- A new ticket should track this work (additive-v2 dual-write).
|
||||
|
||||
## Decided knobs (no longer open)
|
||||
|
||||
- Migration ceremony: **automatic** (upgraded client dual-writes on push; no
|
||||
required `migrate`). `migrate`/`--status` remain manual/inspection tools.
|
||||
- Forcing function: **kept**, scoped to brand-new `properties` externals via the
|
||||
twin rule (existing twins always dual-written).
|
||||
- `finalize`: **kept** as optional GC (not removed), so a fully-upgraded store
|
||||
can still be reclaimed to pure v2.
|
||||
Loading…
Add table
Add a link
Reference in a new issue