feat: store-format-v2 self-describing migration (EGB-703, folds in EGB-700)

Stage 2 of EGB-677. Makes the store self-describing and unifies the legacy
external `properties` blob suffix, via a copy-forward migration that never
destroys data until an explicit, gated finalize.

Scope decision (see eureka): the EGB-677 CEO plan's "flatten dotenv blobs to
basename" was dropped as LOSSY — it discards the restore relpath that makes
the store self-describing and adds basename collisions. Engineering analysis
(4 parallel design agents) showed the store is already relpath-self-describing;
the only real v1→v2 delta is the `properties` suffix. This implements the
minimal, safe v2 that achieves the epic's self-describing goal.

What's added:
- `.secrets-format` marker (committed, one line `2`). Absence ⇒ v1 (every
  pre-EGB-703 store). `_store_format()` reads it; `init` stamps fresh stores
  born-v2. `secrets which` prints `format: vN` (EGB-700 folded in).
- `_external_blob_suffix(type)` — single source of truth for the external
  suffix (v2: gradle-properties → properties; file unchanged). push/pull/verify
  all route through it, so v1 and v2 stores never disagree on blob location.
- `secrets migrate` — per-project copy-forward (writes `.properties.age` twins
  beside v1 blobs; idempotent; needs the project manifest), `--dry-run`
  (reports old→new, writes nothing), `--finalize` (store-wide, the only
  destructive step: gates on `verify --all` green + every v1 blob twinned,
  cuts a `pre-v2-migrate-<sha>` recovery tag, stamps the marker, then drops v1
  blobs; refuses without `--yes`/operator confirmation).

rekey and verify --all stay format-agnostic (recursive find walk) — no change.

21 new bats tests (test/migrate.bats): marker/born-v2, format-aware suffix,
v1 back-compat, dry-run, copy-forward idempotency, no-manifest die, finalize
gates (verify-not-green refusal, untwinned refusal, recovery tag, confirmation),
and full v1→window→finalize round-trip. Updated 4 existing tests for the
born-v2 suffix. Full suite 231/231, bash 3.2 clean.
This commit is contained in:
Brian Majewski 2026-06-07 16:07:31 -07:00
parent 144ff3692b
commit e2ad661da5
6 changed files with 487 additions and 10 deletions

View file

@ -56,12 +56,13 @@ skips security specialist + red team, and Step 11 skips adversarial review.
## Architecture ## Architecture
Single bash script (`secrets`) with subcommands: init, push, pull, list, rm, rekey, verify. Single bash script (`secrets`) with subcommands: init, push, pull, list, rm, rekey, verify, migrate.
- Encryption: `age` with key files (not passphrases — age passphrases are non-scriptable) - Encryption: `age` with key files (not passphrases — age passphrases are non-scriptable)
- Storage: Private git repo at `~/.secrets/` - Storage: Private git repo at `~/.secrets/`
- Convention: Tracks `.env`, `.env.*`, and `.dev.vars` (not `.envrc`, `.environment-*`) - Convention: Tracks `.env`, `.env.*`, and `.dev.vars` (not `.envrc`, `.environment-*`)
- Manifest (EGB-677 stage 1): committed `.secrets.json` is the source of truth for what syncs — `dotenv[]` (project-relative, nested ok, `@` allowed; rail rejects `..`/absolute/symlink) + `external[]` (`properties`/`file`). Push discovery auto-adds (gated by committed `options.autoAdd`, default ON; `--frozen`/`--dry-run` overrides), bootstraps the manifest on first push (written only after ≥1 blob encrypts), and absorbs a legacy `.secrets-files` (gradle-properties → `properties`; on pull the legacy file is superseded with a warning). v1 store layout unchanged in stage 1: nested entries land at `<project>/<relpath>.age`; `properties` blobs keep the legacy `.gradle-properties.age` suffix until the stage-2 store migration. jq is a hard dep only when a manifest exists/is written; manifest-less projects run jq-free (manifest features skipped with a notice). `check_cmd` prints platform-aware install hints. - Manifest (EGB-677 stage 1): committed `.secrets.json` is the source of truth for what syncs — `dotenv[]` (project-relative, nested ok, `@` allowed; rail rejects `..`/absolute/symlink) + `external[]` (`properties`/`file`). Push discovery auto-adds (gated by committed `options.autoAdd`, default ON; `--frozen`/`--dry-run` overrides), bootstraps the manifest on first push (written only after ≥1 blob encrypts), and absorbs a legacy `.secrets-files` (gradle-properties → `properties`; on pull the legacy file is superseded with a warning). Store layout: nested dotenv entries land at `<project>/<relpath>.age` (relpath preserved — the store self-describes where a file restores). jq is a hard dep only when a manifest exists/is written; manifest-less projects run jq-free (manifest features skipped with a notice). `check_cmd` prints platform-aware install hints.
- Store format (EGB-677 stage 2 / EGB-703): the store is self-describing via a committed one-line `$SECRETS_DIR/.secrets-format` file (`2`). Absence ⇒ v1 (every store predating EGB-703). v2's only on-disk change vs v1 is the external `properties` blob suffix: `.gradle-properties.age``.properties.age` (matching the manifest `type`); dotenv and `file` blobs are unchanged. `_store_format()` reads the marker; `_external_blob_suffix(type)` is the single source of truth for the suffix (push/pull/verify all route through it, so v1 and v2 stores never disagree on where a blob lives). `init` stamps a fresh store v2 (born-v2). `secrets which` prints `format: vN`. **Migration is copy-forward and non-destructive:** `secrets migrate --dry-run` (per project, reports old→new, writes nothing) → `secrets migrate` (per project, writes `.properties.age` twins beside the v1 blobs; needs the project manifest to know which externals are `properties`; idempotent) → `secrets migrate --finalize` (store-wide; the ONLY destructive step — gates on `verify --all` green + every v1 blob having a v2 twin, cuts a `pre-v2-migrate-<sha>` recovery tag, stamps the marker, then drops v1 blobs; refuses without `--yes`/operator confirmation since a lagging v1 client against a finalized store stops seeing `properties` externals until it upgrades). The deliberate flatten-to-basename naming the EGB-677 CEO plan sketched was dropped as lossy (it discards the restore relpath that makes the store self-describing) — see the EGB-703 eureka.
- Verify (EGB-698): `secrets verify` is a read-only integrity check. Default mode (current project) cross-checks `$PWD/.secrets.json` against `$SECRETS_DIR/<project>/` both ways (declared-but-missing blobs + orphaned blobs) and decrypt-tests every blob (dotenv + external) by streaming plaintext to `/dev/null` (never written to disk). `secrets verify --all` decrypt-tests every blob in every project (integrity only — the store carries no manifests, so consistency can't be checked store-wide). Both recurse the whole project tree (`find -type f`, same as rekey/list). Exits non-zero on any finding so it can gate the stage-2 `migrate --finalize` and CI. The store deliberately holds no manifest — `.secrets.json` is committed in each project's own repo and read from `$PWD`. - Verify (EGB-698): `secrets verify` is a read-only integrity check. Default mode (current project) cross-checks `$PWD/.secrets.json` against `$SECRETS_DIR/<project>/` both ways (declared-but-missing blobs + orphaned blobs) and decrypt-tests every blob (dotenv + external) by streaming plaintext to `/dev/null` (never written to disk). `secrets verify --all` decrypt-tests every blob in every project (integrity only — the store carries no manifests, so consistency can't be checked store-wide). Both recurse the whole project tree (`find -type f`, same as rekey/list). Exits non-zero on any finding so it can gate the stage-2 `migrate --finalize` and CI. The store deliberately holds no manifest — `.secrets.json` is committed in each project's own repo and read from `$PWD`.
- External files: `.secrets-files` manifest tracks designated keys from files outside the project (e.g. `~/.gradle/gradle.properties`, merged not overwritten — EGB-531) and whole binary files (type `file`, e.g. an Android upload keystore — EGB-652); see below - External files: `.secrets-files` manifest tracks designated keys from files outside the project (e.g. `~/.gradle/gradle.properties`, merged not overwritten — EGB-531) and whole binary files (type `file`, e.g. an Android upload keystore — EGB-652); see below
- Workspaces: `--workspaces` flag reads `package.json` workspaces, requires `jq` - Workspaces: `--workspaces` flag reads `package.json` workspaces, requires `jq`

View file

@ -171,6 +171,8 @@ secrets clear
| `secrets rekey` | Generate a new encryption key and re-encrypt everything | | `secrets rekey` | Generate a new encryption key and re-encrypt everything |
| `secrets verify [project]` | Check the current project's `.secrets.json` against the store (missing/orphaned blobs) and decrypt every blob. `[project]` overrides the store directory name; the manifest is still read from the current directory | | `secrets verify [project]` | Check the current project's `.secrets.json` against the store (missing/orphaned blobs) and decrypt every blob. `[project]` overrides the store directory name; the manifest is still read from the current directory |
| `secrets verify --all` | Decrypt-test every blob in every project — a store-wide integrity sweep | | `secrets verify --all` | Decrypt-test every blob in every project — a store-wide integrity sweep |
| `secrets migrate [--dry-run]` | Copy-forward this project's encrypted blobs to store format v2 (non-destructive; `--dry-run` previews) |
| `secrets migrate --finalize` | Drop the old v1 blobs and mark the store v2 — runs once, store-wide, after `verify` is green and every machine is upgraded |
### Automatic project detection ### Automatic project detection

212
secrets
View file

@ -517,6 +517,39 @@ _secrets_files_slug() {
printf '%s-%s' "$clean" "$sum" printf '%s-%s' "$clean" "$sum"
} }
# ─── Store format (EGB-703) ───────────────────────────────────────────
#
# The store self-describes its format via a committed one-line file
# `$SECRETS_DIR/.secrets-format` containing `2`. Absence (or any non-`2`
# content) means format v1 — the legacy default for every store that
# predates EGB-703. `init` stamps a fresh store v2 (born-v2); `migrate
# --finalize` stamps a migrated store v2. Requires resolve_store to have
# run (SECRETS_DIR set).
STORE_FORMAT_FILE_NAME=".secrets-format"
_store_format() {
local f="$SECRETS_DIR/$STORE_FORMAT_FILE_NAME" v
if [ -f "$f" ]; then
v=$(head -1 "$f" 2>/dev/null | tr -dc '0-9')
[ "$v" = "2" ] && { echo 2; return; }
fi
echo 1
}
# The on-disk blob suffix for an external entry, format-aware. v2 unifies
# the legacy `gradle-properties` suffix to `properties` (matching the JSON
# manifest `type`); `file` is unchanged in both formats. The slug + this
# suffix + `.age` is the external blob name. This is the single source of
# truth for the suffix — push, pull, verify all route through it so a v1
# and a v2 store can never disagree on where a blob lives.
_external_blob_suffix() {
local mtype="$1"
if [ "$mtype" = "gradle-properties" ] && [ "$(_store_format)" = "2" ]; then
echo "properties"
else
echo "$mtype"
fi
}
# Merge managed key=value lines (from $2) into target file $1, preserving # Merge managed key=value lines (from $2) into target file $1, preserving
# all unrelated lines/comments/order. Updates a managed key in place (first # all unrelated lines/comments/order. Updates a managed key in place (first
# occurrence), collapses duplicates, appends new keys. Atomic + mode-safe. # occurrence), collapses duplicates, appends new keys. Atomic + mode-safe.
@ -615,7 +648,7 @@ push_external_files() {
# EGB-652: whole-file sync — encrypt the file verbatim (binary-safe). # EGB-652: whole-file sync — encrypt the file verbatim (binary-safe).
mkdir -p "$SECRETS_DIR/$project/external" mkdir -p "$SECRETS_DIR/$project/external"
local fslug; fslug=$(_secrets_files_slug "$mpath") local fslug; fslug=$(_secrets_files_slug "$mpath")
age -r "$pubkey" -o "$SECRETS_DIR/$project/external/$fslug.file.age" "$expanded" age -r "$pubkey" -o "$SECRETS_DIR/$project/external/$fslug.$(_external_blob_suffix file).age" "$expanded"
info "Encrypted file $mpath" info "Encrypted file $mpath"
pushed=$((pushed + 1)) pushed=$((pushed + 1))
continue continue
@ -644,7 +677,7 @@ push_external_files() {
fi fi
mkdir -p "$SECRETS_DIR/$project/external" mkdir -p "$SECRETS_DIR/$project/external"
local slug; slug=$(_secrets_files_slug "$mpath") local slug; slug=$(_secrets_files_slug "$mpath")
age -r "$pubkey" -o "$SECRETS_DIR/$project/external/$slug.$mtype.age" "$tmp" age -r "$pubkey" -o "$SECRETS_DIR/$project/external/$slug.$(_external_blob_suffix "$mtype").age" "$tmp"
rm -f "$tmp" rm -f "$tmp"
info "Extracted $found key(s) from $mpath" info "Extracted $found key(s) from $mpath"
pushed=$((pushed + 1)) pushed=$((pushed + 1))
@ -673,7 +706,7 @@ pull_external_files() {
continue continue
fi fi
local slug; slug=$(_secrets_files_slug "$mpath") local slug; slug=$(_secrets_files_slug "$mpath")
local blob="$SECRETS_DIR/$project/external/$slug.$mtype.age" local blob="$SECRETS_DIR/$project/external/$slug.$(_external_blob_suffix "$mtype").age"
if [ ! -f "$blob" ]; then if [ ! -f "$blob" ]; then
echo "WARNING: $SECRETS_FILES_NAME names '$mpath' but no encrypted data exists in the store yet. Run 'secrets push' on a machine that has these keys. Skipping." >&2 echo "WARNING: $SECRETS_FILES_NAME names '$mpath' but no encrypted data exists in the store yet. Run 'secrets push' on a machine that has these keys. Skipping." >&2
continue continue
@ -1126,6 +1159,11 @@ Your key file has been left untouched."
# Write .gitignore # Write .gitignore
write_store_gitignore write_store_gitignore
# Stamp the store format (EGB-703): a fresh store is born v2 — it has no
# v1 blobs, so it is already in v2 shape. The marker is a committed,
# non-secret metadata file (NOT gitignored); the first push stages it.
printf '2\n' > "$SECRETS_DIR/$STORE_FORMAT_FILE_NAME"
# Install pre-commit hook # Install pre-commit hook
mkdir -p "$SECRETS_DIR/.git/hooks" mkdir -p "$SECRETS_DIR/.git/hooks"
install_hook install_hook
@ -1866,6 +1904,9 @@ cmd_which() {
resolve_store resolve_store
echo "store: $SECRETS_DIR" echo "store: $SECRETS_DIR"
echo "source: $STORE_SOURCE" echo "source: $STORE_SOURCE"
# EGB-700: surface the store format so users can tell v1 from v2 during the
# migration window. A v1 store is a legacy store with no format marker.
echo "format: v$(_store_format)"
# v2 manifest (.secrets.json): validate and summarize. Validation here # v2 manifest (.secrets.json): validate and summarize. Validation here
# is deliberately fatal (symlink / malformed / unsupported version) so # is deliberately fatal (symlink / malformed / unsupported version) so
@ -2005,7 +2046,7 @@ _verify_project() {
while IFS=$'\t' read -r etype epath _; do while IFS=$'\t' read -r etype epath _; do
[ -n "$etype" ] || continue [ -n "$etype" ] || continue
slug=$(_secrets_files_slug "$epath") slug=$(_secrets_files_slug "$epath")
erel="external/$slug.$etype.age" erel="external/$slug.$(_external_blob_suffix "$etype").age"
expected="$expected$erel"$'\n' expected="$expected$erel"$'\n'
eblob="$pdir/$erel" eblob="$pdir/$erel"
if [ ! -f "$eblob" ]; then if [ ! -f "$eblob" ]; then
@ -2066,6 +2107,166 @@ cmd_verify() {
fi fi
} }
# ─── Store-format-v2 migration (EGB-703) ──────────────────────────────
#
# v2 renames the legacy `properties` blob suffix (.gradle-properties.age →
# .properties.age) and marks the store self-describing via .secrets-format.
# Migration is copy-forward and non-destructive until --finalize:
# secrets migrate --dry-run # per project: report old→new, write nothing
# secrets migrate # per project: write v2 twins beside v1 blobs
# secrets migrate --finalize # store-wide: verify, drop v1, stamp v2
# Per-project (needs the project manifest to know which externals are
# `properties`); finalize is store-wide. Mirrors verify's project/--all split.
# Copy-forward (or dry-run preview) for the current project. Reads
# $PWD/.secrets.json; only `properties` external blobs rename in v2.
_migrate_project() {
local dry_run="$1"
if [ "$(_store_format)" = "2" ]; then
info "Store is already format v2 — nothing to migrate."
return 0
fi
local manifest="$PWD/$SECRETS_JSON_NAME"
if [ ! -e "$manifest" ]; then
die "No $SECRETS_JSON_NAME in $PWD.
'secrets migrate' copy-forwards a project's v1 blobs to their v2 names and
reads the project manifest to do so. cd into a project that has a manifest,
then run 'secrets migrate'. (Store-wide 'secrets migrate --finalize' comes
after every project is migrated.)"
fi
_check_manifest_file "$manifest"
local project; project=$(derive_project_name "")
local pdir="$SECRETS_DIR/$project"
local moved=0 already=0 would=0 etype epath slug old new
while IFS=$'\t' read -r etype epath _; do
[ -n "$etype" ] || continue
# Only `properties` blobs change name in v2; dotenv and `file` are already
# in their v2 shape and never move.
[ "$etype" = "gradle-properties" ] || continue
slug=$(_secrets_files_slug "$epath")
old="$pdir/external/$slug.gradle-properties.age"
new="$pdir/external/$slug.properties.age"
[ -f "$old" ] || continue # nothing pushed yet (or already dropped)
if [ -f "$new" ]; then # idempotent: twin already exists
already=$((already + 1))
continue
fi
if [ "$dry_run" = true ]; then
echo "would migrate: $project/external/$slug.gradle-properties.age -> $slug.properties.age"
would=$((would + 1))
else
cp "$old" "$new"
moved=$((moved + 1))
fi
done < <(_json_external_entries "$manifest")
if [ "$dry_run" = true ]; then
echo "migrate --dry-run: $would blob(s) would be copy-forwarded for '$project' (writes nothing); $already already present. v1 blobs are kept until 'secrets migrate --finalize'."
return 0
fi
if [ "$moved" -eq 0 ] && [ "$already" -eq 0 ]; then
info "Nothing to migrate for '$project' (no v1 properties blobs)."
return 0
fi
ensure_store_protections
git -C "$SECRETS_DIR" add -A
git -C "$SECRETS_DIR" commit -m "migrate: copy-forward v2 twins for $project" >/dev/null 2>&1 || true
info "Copy-forward for '$project': $moved new v2 twin(s), $already already present. v1 blobs kept (non-destructive). Run 'secrets migrate --finalize' once every project is migrated and every machine is upgraded."
}
# Store-wide finalize: the only destructive step. Refuses unless verify --all
# is green and every v1 properties blob has a v2 twin. Cuts a recovery tag,
# stamps the marker, then drops v1 blobs.
_migrate_finalize() {
local force="$1"
check_key
if [ "$(_store_format)" = "2" ]; then
info "Store is already format v2 — nothing to finalize."
return 0
fi
# (gate 1) every blob must decrypt with the current key.
info "Verifying every blob decrypts before finalizing..."
if ! _verify_all >/dev/null 2>&1; then
die "Refusing to finalize: 'secrets verify --all' is not green — a blob does not decrypt. Run 'secrets verify --all' to see which, fix it, then re-run --finalize."
fi
# (gate 2) every v1 properties blob must have a v2 twin (project migrated).
local untwinned="" f new v1count=0
while IFS= read -r f; do
[ -f "$f" ] || continue
v1count=$((v1count + 1))
new="${f%.gradle-properties.age}.properties.age"
[ -f "$new" ] || untwinned="$untwinned ${f#"$SECRETS_DIR"/}"$'\n'
done < <(find "$SECRETS_DIR" -type f -name '*.gradle-properties.age')
if [ -n "$untwinned" ]; then
die "Refusing to finalize: these v1 properties blobs have no v2 twin (their project was not migrated):
$untwinned cd into each project and run 'secrets migrate', then re-run 'secrets migrate --finalize'."
fi
if [ "$v1count" -eq 0 ]; then
# No v1 blobs at all — just stamp the marker (dotenv/file-only store).
printf '2\n' > "$SECRETS_DIR/$STORE_FORMAT_FILE_NAME"
ensure_store_protections
git -C "$SECRETS_DIR" add -A
git -C "$SECRETS_DIR" commit -m "migrate: finalize store format v2" >/dev/null 2>&1 || true
git -C "$SECRETS_DIR" remote get-url origin >/dev/null 2>&1 && git -C "$SECRETS_DIR" push >/dev/null 2>&1 || true
info "Store finalized to format v2 (no v1 blobs to drop)."
return 0
fi
# (gate 3) operator confirms every machine is upgraded.
if [ "$force" != true ]; then
echo "Finalize will drop $v1count v1 blob(s) and stamp the store format v2."
echo "Any machine still running a pre-v2 'secrets' will stop seeing 'properties'"
echo "externals from this store until it upgrades (git pull in the secrets repo)."
printf "Confirm every machine is upgraded? Type 'yes': "
local reply=""
read -r reply < /dev/tty 2>/dev/null || read -r reply || true
[ "$reply" = "yes" ] || die "Finalize aborted — no confirmation."
fi
# Recovery tag BEFORE any mutation: points at the pre-finalize commit (v1
# blobs intact, no marker), so `git checkout <tag>` fully restores v1.
local tag="pre-v2-migrate-$(git -C "$SECRETS_DIR" rev-parse --short HEAD 2>/dev/null || echo unknown)"
git -C "$SECRETS_DIR" tag "$tag" >/dev/null 2>&1 || true
# Stamp the marker FIRST, then drop v1 blobs. If finalize crashes between
# the two, the store reads as v2 and the (verified) v2 twins serve every
# upgraded client; leftover v1 blobs are harmless orphans a re-run cleans.
printf '2\n' > "$SECRETS_DIR/$STORE_FORMAT_FILE_NAME"
while IFS= read -r f; do
[ -f "$f" ] || continue
rm -f "$f"
done < <(find "$SECRETS_DIR" -type f -name '*.gradle-properties.age')
ensure_store_protections
git -C "$SECRETS_DIR" add -A
git -C "$SECRETS_DIR" commit -m "migrate: finalize store format v2 (drop $v1count v1 blob(s))" >/dev/null 2>&1 || true
git -C "$SECRETS_DIR" remote get-url origin >/dev/null 2>&1 && git -C "$SECRETS_DIR" push >/dev/null 2>&1 || true
info "Store finalized to format v2. Dropped $v1count v1 blob(s). Recovery tag in the store: $tag"
}
cmd_migrate() {
resolve_store
check_initialized
local dry_run=false finalize=false force=false
while [ $# -gt 0 ]; do
case "$1" in
--dry-run) dry_run=true; shift ;;
--finalize) finalize=true; shift ;;
--yes|--force) force=true; shift ;;
-*) die "Unknown migrate flag: $1. Usage: secrets migrate [--dry-run | --finalize] [--yes]" ;;
*) die "migrate takes no project argument. Run it from inside a project (copy-forward) or use --finalize (store-wide)." ;;
esac
done
if [ "$finalize" = true ]; then
_migrate_finalize "$force"
else
_migrate_project "$dry_run"
fi
}
cmd_help() { cmd_help() {
cat << 'EOF' cat << 'EOF'
secrets — encrypted secret file sync between machines secrets — encrypted secret file sync between machines
@ -2087,6 +2288,8 @@ Usage:
secrets rekey Re-encrypt all secrets with a new key secrets rekey Re-encrypt all secrets with a new key
secrets verify [project] Check the manifest against the store + decrypt every blob secrets verify [project] Check the manifest against the store + decrypt every blob
secrets verify --all Decrypt-test every blob in every project (integrity gate) secrets verify --all Decrypt-test every blob in every project (integrity gate)
secrets migrate [--dry-run] Copy-forward this project's blobs to store format v2
secrets migrate --finalize Drop v1 blobs and mark the store v2 (after verify)
secrets which Show the active store, manifest, and external entries secrets which Show the active store, manifest, and external entries
secrets where Alias for `which` secrets where Alias for `which`
secrets status Alias for `which` secrets status Alias for `which`
@ -2256,6 +2459,7 @@ case "${1:-help}" in
rm) cmd_rm "${2:-}" ;; rm) cmd_rm "${2:-}" ;;
rekey) cmd_rekey ;; rekey) cmd_rekey ;;
verify) shift; cmd_verify "$@" ;; verify) shift; cmd_verify "$@" ;;
migrate) shift; cmd_migrate "$@" ;;
which|where|status) cmd_which ;; which|where|status) cmd_which ;;
help|--help|-h) cmd_help ;; help|--help|-h) cmd_help ;;
*) die "Unknown command: $1. Run 'secrets help' for usage." ;; *) die "Unknown command: $1. Run 'secrets help' for usage." ;;

View file

@ -269,7 +269,7 @@ m_file_src() { mkdir -p "$HOME/keystores"; printf 'KS\x00\x01\x02\xffDATA\n' >
run jq -r '.external[] | select(.type == "file") | .path' .secrets.json run jq -r '.external[] | select(.type == "file") | .path' .secrets.json
[ "$output" = "~/keystores/upload.keystore" ] [ "$output" = "~/keystores/upload.keystore" ]
# stage 1: blob naming stays legacy-compatible # stage 1: blob naming stays legacy-compatible
run bash -c "ls $SECRETS_DIR/absorbproj/external/*.gradle-properties.age" run bash -c "ls $SECRETS_DIR/absorbproj/external/*.properties.age"
[ "$status" -eq 0 ] [ "$status" -eq 0 ]
} }
@ -296,7 +296,7 @@ m_file_src() { mkdir -p "$HOME/keystores"; printf 'KS\x00\x01\x02\xffDATA\n' >
run "$SECRETS_BIN" push jsonextproj run "$SECRETS_BIN" push jsonextproj
[ "$status" -eq 0 ] [ "$status" -eq 0 ]
[[ "$output" == *"Extracted 1 key"* ]] || false [[ "$output" == *"Extracted 1 key"* ]] || false
run bash -c "ls $SECRETS_DIR/jsonextproj/external/*.gradle-properties.age" run bash -c "ls $SECRETS_DIR/jsonextproj/external/*.properties.age"
[ "$status" -eq 0 ] [ "$status" -eq 0 ]
} }

270
test/migrate.bats Normal file
View file

@ -0,0 +1,270 @@
#!/usr/bin/env bats
# EGB-703 store-format-v2: marker, format-aware suffix, migrate (dry-run /
# copy-forward / finalize). bash 3.2: every standalone [[ ]] ends with || false.
load test_helper
# A v1 (legacy) store: born-v2 init, then strip the marker so it reads as v1
# and pushes write the legacy .gradle-properties.age suffix.
make_v1_store() {
init_with_remote
rm -f "$SECRETS_DIR/.secrets-format"
}
m_gradle_src() { mkdir -p "$HOME/.gradle"; printf '%s' "$1" > "$HOME/.gradle/gradle.properties"; }
m_file_src() { mkdir -p "$HOME/keystores"; printf 'KS\x00\x01\x02\xffDATA\n' > "$HOME/keystores/upload.keystore"; }
# ─── Format marker + format-aware suffix (increment 1) ────────────────
@test "init stamps the store format marker as v2 (born-v2)" {
init_with_remote
[ -f "$SECRETS_DIR/.secrets-format" ]
[ "$(cat "$SECRETS_DIR/.secrets-format")" = "2" ]
}
@test "which prints format v2 for a born-v2 store" {
init_with_remote
create_project_dir whichv2
run "$SECRETS_BIN" which
[ "$status" -eq 0 ]
[[ "$output" == *"format: v2"* ]] || false
}
@test "which prints format v1 for a markerless (legacy) store" {
make_v1_store
create_project_dir whichv1
run "$SECRETS_BIN" which
[ "$status" -eq 0 ]
[[ "$output" == *"format: v1"* ]] || false
}
@test "push on a born-v2 store writes the properties blob as .properties.age" {
init_with_remote
m_gradle_src $'beaconClerkPkTest=pk_test_abc\n'
create_project_dir v2push
printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files
"$SECRETS_BIN" push v2push >/dev/null 2>&1
run bash -c "ls $SECRETS_DIR/v2push/external/*.properties.age"
[ "$status" -eq 0 ]
run bash -c "ls $SECRETS_DIR/v2push/external/*.gradle-properties.age 2>/dev/null"
[ "$status" -ne 0 ]
}
@test "push on a v1 store still writes .gradle-properties.age (back-compat)" {
make_v1_store
m_gradle_src $'beaconClerkPkTest=pk_test_abc\n'
create_project_dir v1push
printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files
"$SECRETS_BIN" push v1push >/dev/null 2>&1
run bash -c "ls $SECRETS_DIR/v1push/external/*.gradle-properties.age"
[ "$status" -eq 0 ]
}
@test "the format marker is committed, not gitignored" {
init_with_remote
create_project_dir markercommit
"$SECRETS_BIN" push markercommit >/dev/null 2>&1
run bash -c "git -C $SECRETS_DIR ls-files | grep -qx .secrets-format"
[ "$status" -eq 0 ]
}
# ─── migrate --dry-run / copy-forward (increment 2) ───────────────────
@test "migrate --dry-run reports the rename and writes nothing" {
make_v1_store
m_gradle_src $'beaconClerkPkTest=pk_test_abc\n'
create_project_dir dryproj
printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files
"$SECRETS_BIN" push dryproj >/dev/null 2>&1
run "$SECRETS_BIN" migrate --dry-run
[ "$status" -eq 0 ]
[[ "$output" == *"would migrate"* ]] || false
# nothing written
run bash -c "ls $SECRETS_DIR/dryproj/external/*.properties.age 2>/dev/null"
[ "$status" -ne 0 ]
# marker still absent (store still v1)
[ ! -f "$SECRETS_DIR/.secrets-format" ]
}
@test "migrate --dry-run on a dotenv-only project reports nothing to migrate" {
make_v1_store
create_project_dir dotenvonly
"$SECRETS_BIN" push dotenvonly >/dev/null 2>&1
run "$SECRETS_BIN" migrate --dry-run
[ "$status" -eq 0 ]
[[ "$output" == *"0 blob(s) would be copy-forwarded"* ]] || false
}
@test "migrate copy-forward creates the v2 twin and keeps the v1 blob (byte-identical)" {
make_v1_store
m_gradle_src $'beaconClerkPkTest=pk_test_abc\n'
create_project_dir cfproj
printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files
"$SECRETS_BIN" push cfproj >/dev/null 2>&1
local old; old=$(ls "$SECRETS_DIR/cfproj/external/"*.gradle-properties.age)
run "$SECRETS_BIN" migrate
[ "$status" -eq 0 ]
local new; new=$(ls "$SECRETS_DIR/cfproj/external/"*.properties.age)
[ -f "$old" ] # v1 kept (non-destructive)
[ -f "$new" ] # v2 twin written
cmp -s "$old" "$new" # byte-identical ciphertext copy
}
@test "migrate copy-forward is idempotent" {
make_v1_store
m_gradle_src $'beaconClerkPkTest=pk_test_abc\n'
create_project_dir idemproj
printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files
"$SECRETS_BIN" push idemproj >/dev/null 2>&1
"$SECRETS_BIN" migrate >/dev/null 2>&1
run "$SECRETS_BIN" migrate
[ "$status" -eq 0 ]
[[ "$output" == *"1 already present"* ]] || false
run bash -c "ls $SECRETS_DIR/idemproj/external/*.properties.age | wc -l | tr -d ' '"
[ "$output" = "1" ]
}
@test "migrate with no manifest in cwd dies with a directed message" {
make_v1_store
local dir="$WORK_DIR/nomanifest"; mkdir -p "$dir"; cd "$dir"
run "$SECRETS_BIN" migrate
[ "$status" -eq 1 ]
[[ "$output" == *".secrets.json"* ]] || false
}
@test "migrate on an already-v2 store is a no-op" {
init_with_remote
create_project_dir alreadyv2
run "$SECRETS_BIN" migrate
[ "$status" -eq 0 ]
[[ "$output" == *"already format v2"* ]] || false
}
@test "migrate unknown flag dies with usage" {
init_with_remote
create_project_dir mgflag
run "$SECRETS_BIN" migrate --bogus
[ "$status" -eq 1 ]
[[ "$output" == *"Unknown migrate flag"* ]] || false
}
@test "migrate leaves dotenv and file blobs untouched" {
make_v1_store
m_gradle_src $'beaconClerkPkTest=pk_test_abc\n'
m_file_src
create_project_dir mixproj
printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\nfile ~/keystores/upload.keystore\n' > .secrets-files
"$SECRETS_BIN" push mixproj >/dev/null 2>&1
local envblob; envblob=$(ls "$SECRETS_DIR/mixproj/".env.age)
local fileblob; fileblob=$(ls "$SECRETS_DIR/mixproj/external/"*.file.age)
local envsum; envsum=$(cksum "$envblob")
local filesum; filesum=$(cksum "$fileblob")
"$SECRETS_BIN" migrate >/dev/null 2>&1
[ "$(cksum "$envblob")" = "$envsum" ] # dotenv blob unchanged
[ "$(cksum "$fileblob")" = "$filesum" ] # file blob unchanged
}
# ─── migrate --finalize (increment 3) ─────────────────────────────────
@test "finalize refuses when verify --all is not green" {
make_v1_store
m_gradle_src $'beaconClerkPkTest=pk_test_abc\n'
create_project_dir failverify
printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files
"$SECRETS_BIN" push failverify >/dev/null 2>&1
"$SECRETS_BIN" migrate >/dev/null 2>&1
# corrupt the v2 twin so verify --all fails
printf 'garbage' > "$SECRETS_DIR/failverify/external/"*.properties.age
run "$SECRETS_BIN" migrate --finalize --yes
[ "$status" -eq 1 ]
[[ "$output" == *"not green"* ]] || false
# marker not stamped; v1 blob still present
[ ! -f "$SECRETS_DIR/.secrets-format" ]
run bash -c "ls $SECRETS_DIR/failverify/external/*.gradle-properties.age"
[ "$status" -eq 0 ]
}
@test "finalize refuses an un-twinned v1 blob (project not migrated)" {
make_v1_store
m_gradle_src $'beaconClerkPkTest=pk_test_abc\n'
create_project_dir untwinned
printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files
"$SECRETS_BIN" push untwinned >/dev/null 2>&1
# do NOT migrate — leave the v1 blob with no twin
run "$SECRETS_BIN" migrate --finalize --yes
[ "$status" -eq 1 ]
[[ "$output" == *"no v2 twin"* ]] || false
run bash -c "ls $SECRETS_DIR/untwinned/external/*.gradle-properties.age"
[ "$status" -eq 0 ]
}
@test "finalize green path drops v1, keeps v2, stamps the marker" {
make_v1_store
m_gradle_src $'beaconClerkPkTest=pk_test_abc\n'
create_project_dir finproj
printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files
"$SECRETS_BIN" push finproj >/dev/null 2>&1
"$SECRETS_BIN" migrate >/dev/null 2>&1
run "$SECRETS_BIN" migrate --finalize --yes
[ "$status" -eq 0 ]
[ "$(cat "$SECRETS_DIR/.secrets-format")" = "2" ]
run bash -c "ls $SECRETS_DIR/finproj/external/*.properties.age"
[ "$status" -eq 0 ]
run bash -c "ls $SECRETS_DIR/finproj/external/*.gradle-properties.age 2>/dev/null"
[ "$status" -ne 0 ]
}
@test "finalize cuts a recovery tag before deleting v1 blobs" {
make_v1_store
m_gradle_src $'beaconClerkPkTest=pk_test_abc\n'
create_project_dir tagproj
printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files
"$SECRETS_BIN" push tagproj >/dev/null 2>&1
"$SECRETS_BIN" migrate >/dev/null 2>&1
"$SECRETS_BIN" migrate --finalize --yes >/dev/null 2>&1
local tag; tag=$(git -C "$SECRETS_DIR" tag | grep '^pre-v2-migrate-')
[ -n "$tag" ]
# the tagged commit still contains the v1 blob (tag cut before delete)
run bash -c "git -C $SECRETS_DIR ls-tree -r --name-only $tag | grep -q gradle-properties.age"
[ "$status" -eq 0 ]
}
@test "finalize without --yes aborts when not confirmed" {
make_v1_store
m_gradle_src $'beaconClerkPkTest=pk_test_abc\n'
create_project_dir confproj
printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files
"$SECRETS_BIN" push confproj >/dev/null 2>&1
"$SECRETS_BIN" migrate >/dev/null 2>&1
run bash -c "echo '' | $SECRETS_BIN migrate --finalize"
[ "$status" -eq 1 ]
[[ "$output" == *"aborted"* ]] || false
[ ! -f "$SECRETS_DIR/.secrets-format" ]
}
@test "v1 client still reads during the migration window (after copy-forward, before finalize)" {
make_v1_store
m_gradle_src $'beaconClerkPkTest=pk_test_abc\n'
create_project_dir windowproj
printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files
"$SECRETS_BIN" push windowproj >/dev/null 2>&1
"$SECRETS_BIN" migrate >/dev/null 2>&1
# store is still v1 (markerless); pull uses the .gradle-properties.age blob
rm "$HOME/.gradle/gradle.properties"
run "$SECRETS_BIN" pull windowproj
[ "$status" -eq 0 ]
grep -q '^beaconClerkPkTest=pk_test_abc$' "$HOME/.gradle/gradle.properties"
}
@test "post-finalize pull reads the v2 blob" {
make_v1_store
m_gradle_src $'beaconClerkPkTest=pk_test_abc\n'
create_project_dir postfin
printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files
"$SECRETS_BIN" push postfin >/dev/null 2>&1
"$SECRETS_BIN" migrate >/dev/null 2>&1
"$SECRETS_BIN" migrate --finalize --yes >/dev/null 2>&1
rm "$HOME/.gradle/gradle.properties"
run "$SECRETS_BIN" pull postfin
[ "$status" -eq 0 ]
grep -q '^beaconClerkPkTest=pk_test_abc$' "$HOME/.gradle/gradle.properties"
}

View file

@ -1081,7 +1081,7 @@ gradle_project() {
run "$SECRETS_BIN" push gproj run "$SECRETS_BIN" push gproj
[ "$status" -eq 0 ] [ "$status" -eq 0 ]
[[ "$output" == *"Extracted 2 key"* ]] || false [[ "$output" == *"Extracted 2 key"* ]] || false
run bash -c "ls $SECRETS_DIR/gproj/external/*.gradle-properties.age" run bash -c "ls $SECRETS_DIR/gproj/external/*.properties.age"
[ "$status" -eq 0 ] [ "$status" -eq 0 ]
} }
@ -1368,7 +1368,7 @@ gradle_project() {
run "$SECRETS_BIN" push -w run "$SECRETS_BIN" push -w
[ "$status" -eq 0 ] [ "$status" -eq 0 ]
# External blob pushed exactly once (not once per workspace) # External blob pushed exactly once (not once per workspace)
run bash -c "ls $SECRETS_DIR/mono/external/*.gradle-properties.age 2>/dev/null | wc -l | tr -d ' '" run bash -c "ls $SECRETS_DIR/mono/external/*.properties.age 2>/dev/null | wc -l | tr -d ' '"
[ "$output" = "1" ] [ "$output" = "1" ]
} }