Merge pull request 'v0.6.0.0 feat: store-format-v2 migration (EGB-703)' (#5) from brian/egb-703-store-format-v2-migration into main

CHANGELOG.md

@@ -5,6 +5,31 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to a four-digit MAJOR.MINOR.PATCH.MICRO version scheme.
 
+## [0.6.0.0] - 2026-06-07
+
+### Added
+
+- **Self-describing store format (v2) + `secrets migrate` (EGB-703)** — the
+  store now records its format in a committed `.secrets-format` file, and
+  `secrets which` prints it (`format: v2`). A fresh `secrets init` creates a
+  v2 store; existing stores read as v1 until migrated.
+- **`secrets migrate`** — copy-forward a project's encrypted blobs to the v2
+  layout. It is non-destructive: the old blobs are kept until you finalize, so
+  a half-migrated store stays fully readable and recoverable. `secrets migrate
+  --dry-run` previews exactly what would change without writing anything.
+- **`secrets migrate --finalize`** — the one destructive step, run once
+  store-wide. It refuses unless `secrets verify` passes and every blob has its
+  new-format twin, cuts a `pre-v2-migrate-*` recovery tag first, then drops the
+  old blobs. It asks for confirmation (or `--yes`) because a machine still on
+  an older `secrets` will stop seeing migrated external files until it updates.
+
+### Changed
+
+- The external `properties` blob is stored as `<name>.properties.age` in a v2
+  store (was `<name>.gradle-properties.age`), matching the manifest `type`.
+  `push`, `pull`, and `verify` pick the right name automatically from the store
+  format, so v1 and v2 stores both keep working during a migration.
+
 ## [0.5.0.0] - 2026-06-07
 
 ### Added

CLAUDE.md

@@ -16,7 +16,7 @@ cd ~/my-project && ./secrets pull # Pull + decrypt .env* files
 
 ```bash
 brew install bats-core
-bats test/          # runs secrets.bats + manifest.bats
+bats test/          # runs secrets.bats + manifest.bats + migrate.bats
 ./test/run-security.sh   # security regression subset + operator sign-off (see below)
 ```
 
@@ -56,12 +56,13 @@ skips security specialist + red team, and Step 11 skips adversarial review.
 
 ## Architecture
 
-Single bash script (`secrets`) with subcommands: init, push, pull, list, rm, rekey, verify.
+Single bash script (`secrets`) with subcommands: init, push, pull, list, rm, rekey, verify, migrate.
 
 - Encryption: `age` with key files (not passphrases — age passphrases are non-scriptable)
 - Storage: Private git repo at `~/.secrets/`
 - Convention: Tracks `.env`, `.env.*`, and `.dev.vars` (not `.envrc`, `.environment-*`)
-- Manifest (EGB-677 stage 1): committed `.secrets.json` is the source of truth for what syncs — `dotenv[]` (project-relative, nested ok, `@` allowed; rail rejects `..`/absolute/symlink) + `external[]` (`properties`/`file`). Push discovery auto-adds (gated by committed `options.autoAdd`, default ON; `--frozen`/`--dry-run` overrides), bootstraps the manifest on first push (written only after ≥1 blob encrypts), and absorbs a legacy `.secrets-files` (gradle-properties → `properties`; on pull the legacy file is superseded with a warning). v1 store layout unchanged in stage 1: nested entries land at `<project>/<relpath>.age`; `properties` blobs keep the legacy `.gradle-properties.age` suffix until the stage-2 store migration. jq is a hard dep only when a manifest exists/is written; manifest-less projects run jq-free (manifest features skipped with a notice). `check_cmd` prints platform-aware install hints.
+- Manifest (EGB-677 stage 1): committed `.secrets.json` is the source of truth for what syncs — `dotenv[]` (project-relative, nested ok, `@` allowed; rail rejects `..`/absolute/symlink) + `external[]` (`properties`/`file`). Push discovery auto-adds (gated by committed `options.autoAdd`, default ON; `--frozen`/`--dry-run` overrides), bootstraps the manifest on first push (written only after ≥1 blob encrypts), and absorbs a legacy `.secrets-files` (gradle-properties → `properties`; on pull the legacy file is superseded with a warning). Store layout: nested dotenv entries land at `<project>/<relpath>.age` (relpath preserved — the store self-describes where a file restores). jq is a hard dep only when a manifest exists/is written; manifest-less projects run jq-free (manifest features skipped with a notice). `check_cmd` prints platform-aware install hints.
+- Store format (EGB-677 stage 2 / EGB-703): the store is self-describing via a committed one-line `$SECRETS_DIR/.secrets-format` file (`2`). Absence ⇒ v1 (every store predating EGB-703). v2's only on-disk change vs v1 is the external `properties` blob suffix: `.gradle-properties.age` → `.properties.age` (matching the manifest `type`); dotenv and `file` blobs are unchanged. `_store_format()` reads the marker; `_external_blob_suffix(type)` is the single source of truth for the suffix (push/pull/verify all route through it, so v1 and v2 stores never disagree on where a blob lives). `init` stamps a fresh store v2 (born-v2). `secrets which` prints `format: vN`. **Migration is copy-forward and non-destructive:** `secrets migrate --dry-run` (per project, reports old→new, writes nothing) → `secrets migrate` (per project, writes `.properties.age` twins beside the v1 blobs; needs the project manifest to know which externals are `properties`; idempotent) → `secrets migrate --finalize` (store-wide; the ONLY destructive step — gates on `verify --all` green + every v1 blob having a v2 twin, cuts a `pre-v2-migrate-<sha>` recovery tag, stamps the marker, then drops v1 blobs; refuses without `--yes`/operator confirmation since a lagging v1 client against a finalized store stops seeing `properties` externals until it upgrades). The deliberate flatten-to-basename naming the EGB-677 CEO plan sketched was dropped as lossy (it discards the restore relpath that makes the store self-describing) — see the EGB-703 eureka.
 - Verify (EGB-698): `secrets verify` is a read-only integrity check. Default mode (current project) cross-checks `$PWD/.secrets.json` against `$SECRETS_DIR/<project>/` both ways (declared-but-missing blobs + orphaned blobs) and decrypt-tests every blob (dotenv + external) by streaming plaintext to `/dev/null` (never written to disk). `secrets verify --all` decrypt-tests every blob in every project (integrity only — the store carries no manifests, so consistency can't be checked store-wide). Both recurse the whole project tree (`find -type f`, same as rekey/list). Exits non-zero on any finding so it can gate the stage-2 `migrate --finalize` and CI. The store deliberately holds no manifest — `.secrets.json` is committed in each project's own repo and read from `$PWD`.
 - External files: `.secrets-files` manifest tracks designated keys from files outside the project (e.g. `~/.gradle/gradle.properties`, merged not overwritten — EGB-531) and whole binary files (type `file`, e.g. an Android upload keystore — EGB-652); see below
 - Workspaces: `--workspaces` flag reads `package.json` workspaces, requires `jq`
@@ -76,6 +77,7 @@ hooks/pre-commit     # Pre-commit hook template
 test/
   secrets.bats       # bats-core test suite (133 tests)
   manifest.bats      # EGB-677 .secrets.json manifest tests (77 tests)
+  migrate.bats       # EGB-703 store-format-v2 migration tests (26 tests)
   test_helper.bash   # Shared setup/teardown
 README.md            # User-facing documentation
 CLAUDE.md            # This file

README.md

@@ -171,6 +171,8 @@ secrets clear
 | `secrets rekey` | Generate a new encryption key and re-encrypt everything |
 | `secrets verify [project]` | Check the current project's `.secrets.json` against the store (missing/orphaned blobs) and decrypt every blob. `[project]` overrides the store directory name; the manifest is still read from the current directory |
 | `secrets verify --all` | Decrypt-test every blob in every project — a store-wide integrity sweep |
+| `secrets migrate [--dry-run]` | Copy-forward this project's encrypted blobs to store format v2 (non-destructive; `--dry-run` previews) |
+| `secrets migrate --finalize` | Drop the old v1 blobs and mark the store v2 — runs once, store-wide, after `verify` is green and every machine is upgraded |
 
 ### Automatic project detection
 
@@ -498,7 +500,7 @@ For complete rotation with no historical exposure, create a fresh `~/.secrets/`
 ## Development
 
 ```bash
-# Run the test suite (210 tests across both files)
+# Run the test suite (236 tests across three files)
 brew install bats-core
 bats test/
 

VERSION

@@ -1 +1 @@
-0.5.0.0
+0.6.0.0

secrets

@@ -517,6 +517,43 @@ _secrets_files_slug() {
   printf '%s-%s' "$clean" "$sum"
 }
 
+# ─── Store format (EGB-703) ───────────────────────────────────────────
+#
+# The store self-describes its format via a committed one-line file
+# `$SECRETS_DIR/.secrets-format` containing `2`. Absence (or any non-`2`
+# content) means format v1 — the legacy default for every store that
+# predates EGB-703. `init` stamps a fresh store v2 (born-v2); `migrate
+# --finalize` stamps a migrated store v2. Requires resolve_store to have
+# run (SECRETS_DIR set).
+STORE_FORMAT_FILE_NAME=".secrets-format"
+_store_format() {
+  local f="$SECRETS_DIR/$STORE_FORMAT_FILE_NAME" v
+  if [ -f "$f" ]; then
+    # Strict exact match (modulo line endings): only a marker of exactly "2"
+    # reads as v2. Anything else (empty, "20", "v2", garbage) ⇒ v1 — the safe
+    # default, since misreading v2-as-v1 only triggers a harmless re-migrate
+    # while v1-as-v2 would mislocate blobs.
+    v=$(head -1 "$f" 2>/dev/null | tr -d '\r\n')
+    [ "$v" = "2" ] && { echo 2; return; }
+  fi
+  echo 1
+}
+
+# The on-disk blob suffix for an external entry, format-aware. v2 unifies
+# the legacy `gradle-properties` suffix to `properties` (matching the JSON
+# manifest `type`); `file` is unchanged in both formats. The slug + this
+# suffix + `.age` is the external blob name. This is the single source of
+# truth for the suffix — push, pull, verify all route through it so a v1
+# and a v2 store can never disagree on where a blob lives.
+_external_blob_suffix() {
+  local mtype="$1"
+  if [ "$mtype" = "gradle-properties" ] && [ "$(_store_format)" = "2" ]; then
+    echo "properties"
+  else
+    echo "$mtype"
+  fi
+}
+
 # Merge managed key=value lines (from $2) into target file $1, preserving
 # all unrelated lines/comments/order. Updates a managed key in place (first
 # occurrence), collapses duplicates, appends new keys. Atomic + mode-safe.
@@ -615,7 +652,7 @@ push_external_files() {
       # EGB-652: whole-file sync — encrypt the file verbatim (binary-safe).
       mkdir -p "$SECRETS_DIR/$project/external"
       local fslug; fslug=$(_secrets_files_slug "$mpath")
-      age -r "$pubkey" -o "$SECRETS_DIR/$project/external/$fslug.file.age" "$expanded"
+      age -r "$pubkey" -o "$SECRETS_DIR/$project/external/$fslug.$(_external_blob_suffix file).age" "$expanded"
       info "Encrypted file $mpath"
       pushed=$((pushed + 1))
       continue
@@ -644,7 +681,7 @@ push_external_files() {
     fi
     mkdir -p "$SECRETS_DIR/$project/external"
     local slug; slug=$(_secrets_files_slug "$mpath")
-    age -r "$pubkey" -o "$SECRETS_DIR/$project/external/$slug.$mtype.age" "$tmp"
+    age -r "$pubkey" -o "$SECRETS_DIR/$project/external/$slug.$(_external_blob_suffix "$mtype").age" "$tmp"
     rm -f "$tmp"
     info "Extracted $found key(s) from $mpath"
     pushed=$((pushed + 1))
@@ -673,7 +710,7 @@ pull_external_files() {
       continue
     fi
     local slug; slug=$(_secrets_files_slug "$mpath")
-    local blob="$SECRETS_DIR/$project/external/$slug.$mtype.age"
+    local blob="$SECRETS_DIR/$project/external/$slug.$(_external_blob_suffix "$mtype").age"
     if [ ! -f "$blob" ]; then
       echo "WARNING: $SECRETS_FILES_NAME names '$mpath' but no encrypted data exists in the store yet. Run 'secrets push' on a machine that has these keys. Skipping." >&2
       continue
@@ -1126,6 +1163,11 @@ Your key file has been left untouched."
   # Write .gitignore
   write_store_gitignore
 
+  # Stamp the store format (EGB-703): a fresh store is born v2 — it has no
+  # v1 blobs, so it is already in v2 shape. The marker is a committed,
+  # non-secret metadata file (NOT gitignored); the first push stages it.
+  printf '2\n' > "$SECRETS_DIR/$STORE_FORMAT_FILE_NAME"
+
   # Install pre-commit hook
   mkdir -p "$SECRETS_DIR/.git/hooks"
   install_hook
@@ -1866,6 +1908,9 @@ cmd_which() {
   resolve_store
   echo "store:  $SECRETS_DIR"
   echo "source: $STORE_SOURCE"
+  # EGB-700 (folded into EGB-703): surface the store format so users can tell
+  # v1 from v2 during the migration window. v1 = legacy store, no format marker.
+  echo "format: v$(_store_format)"
 
   # v2 manifest (.secrets.json): validate and summarize. Validation here
   # is deliberately fatal (symlink / malformed / unsupported version) so
@@ -2005,8 +2050,13 @@ _verify_project() {
   while IFS=$'\t' read -r etype epath _; do
     [ -n "$etype" ] || continue
     slug=$(_secrets_files_slug "$epath")
-    erel="external/$slug.$etype.age"
+    erel="external/$slug.$(_external_blob_suffix "$etype").age"
-    expected="$expected$erel"$'\n'
+    # Account for BOTH the v1 and v2 suffix forms in the orphan set. During the
+    # migration window (after copy-forward, before --finalize) the v2 twin
+    # coexists with the v1 blob; neither should read as an orphan whichever
+    # format the store currently reports. (file's two forms are identical.)
+    expected="${expected}external/$slug.$etype.age"$'\n'
+    [ "$etype" = "gradle-properties" ] && expected="${expected}external/$slug.properties.age"$'\n'
     eblob="$pdir/$erel"
     if [ ! -f "$eblob" ]; then
       echo "FINDING: external '$epath' ($etype) is declared but has no blob in the store ($project/$erel missing). Run 'secrets push'." >&2
@@ -2066,6 +2116,171 @@ cmd_verify() {
   fi
 }
 
+# ─── Store-format-v2 migration (EGB-703) ──────────────────────────────
+#
+# v2 renames the legacy `properties` blob suffix (.gradle-properties.age →
+# .properties.age) and marks the store self-describing via .secrets-format.
+# Migration is copy-forward and non-destructive until --finalize:
+#   secrets migrate --dry-run   # per project: report old→new, write nothing
+#   secrets migrate             # per project: write v2 twins beside v1 blobs
+#   secrets migrate --finalize  # store-wide: verify, drop v1, stamp v2
+# Per-project (needs the project manifest to know which externals are
+# `properties`); finalize is store-wide. Mirrors verify's project/--all split.
+
+# Copy-forward (or dry-run preview) for the current project. Reads
+# $PWD/.secrets.json; only `properties` external blobs rename in v2.
+_migrate_project() {
+  local dry_run="$1"
+  if [ "$(_store_format)" = "2" ]; then
+    info "Store is already format v2 — nothing to migrate."
+    return 0
+  fi
+  local manifest="$PWD/$SECRETS_JSON_NAME"
+  if [ ! -e "$manifest" ]; then
+    die "No $SECRETS_JSON_NAME in $PWD.
+  'secrets migrate' copy-forwards a project's v1 blobs to their v2 names and
+  reads the project manifest to do so. cd into a project that has a manifest,
+  then run 'secrets migrate'. (Store-wide 'secrets migrate --finalize' comes
+  after every project is migrated.)"
+  fi
+  _check_manifest_file "$manifest"
+  local project; project=$(derive_project_name "")
+  local pdir="$SECRETS_DIR/$project"
+
+  local moved=0 already=0 would=0 etype epath slug old new
+  while IFS=$'\t' read -r etype epath _; do
+    [ -n "$etype" ] || continue
+    # Only `properties` blobs change name in v2; dotenv and `file` are already
+    # in their v2 shape and never move.
+    [ "$etype" = "gradle-properties" ] || continue
+    slug=$(_secrets_files_slug "$epath")
+    old="$pdir/external/$slug.gradle-properties.age"
+    new="$pdir/external/$slug.properties.age"
+    [ -f "$old" ] || continue          # nothing pushed yet (or already dropped)
+    if [ -f "$new" ]; then             # idempotent: twin already exists
+      already=$((already + 1))
+      continue
+    fi
+    if [ "$dry_run" = true ]; then
+      echo "would migrate: $project/external/$slug.gradle-properties.age -> $slug.properties.age"
+      would=$((would + 1))
+    else
+      cp "$old" "$new"
+      moved=$((moved + 1))
+    fi
+  done < <(_json_external_entries "$manifest")
+
+  if [ "$dry_run" = true ]; then
+    echo "migrate --dry-run: $would blob(s) would be copy-forwarded for '$project' (writes nothing); $already already present. v1 blobs are kept until 'secrets migrate --finalize'."
+    return 0
+  fi
+  if [ "$moved" -eq 0 ] && [ "$already" -eq 0 ]; then
+    info "Nothing to migrate for '$project' (no v1 properties blobs)."
+    return 0
+  fi
+  ensure_store_protections
+  git -C "$SECRETS_DIR" add -A
+  git -C "$SECRETS_DIR" commit -m "migrate: copy-forward v2 twins for $project" >/dev/null 2>&1 || true
+  # Push the twins so a --finalize on another machine sees them (finalize
+  # refuses any v1 blob without a twin). Mirrors push/rekey's push behavior.
+  git -C "$SECRETS_DIR" remote get-url origin >/dev/null 2>&1 && git -C "$SECRETS_DIR" push >/dev/null 2>&1 || true
+  info "Copy-forward for '$project': $moved new v2 twin(s), $already already present. v1 blobs kept (non-destructive). Run 'secrets migrate --finalize' once every project is migrated and every machine is upgraded."
+}
+
+# Store-wide finalize: the only destructive step. Refuses unless verify --all
+# is green and every v1 properties blob has a v2 twin. Cuts a recovery tag,
+# stamps the marker, then drops v1 blobs.
+_migrate_finalize() {
+  local force="$1"
+  check_key
+  if [ "$(_store_format)" = "2" ]; then
+    info "Store is already format v2 — nothing to finalize."
+    return 0
+  fi
+
+  # (gate 1) every blob must decrypt with the current key.
+  info "Verifying every blob decrypts before finalizing..."
+  if ! _verify_all >/dev/null 2>&1; then
+    die "Refusing to finalize: 'secrets verify --all' is not green — a blob does not decrypt. Run 'secrets verify --all' to see which, fix it, then re-run --finalize."
+  fi
+
+  # (gate 2) every v1 properties blob must have a v2 twin (project migrated).
+  local untwinned="" f new v1count=0
+  while IFS= read -r f; do
+    [ -f "$f" ] || continue
+    v1count=$((v1count + 1))
+    new="${f%.gradle-properties.age}.properties.age"
+    [ -f "$new" ] || untwinned="$untwinned  ${f#"$SECRETS_DIR"/}"$'\n'
+  done < <(find "$SECRETS_DIR" -type f -name '*.gradle-properties.age')
+  if [ -n "$untwinned" ]; then
+    die "Refusing to finalize: these v1 properties blobs have no v2 twin (their project was not migrated):
+$untwinned  cd into each project and run 'secrets migrate', then re-run 'secrets migrate --finalize'."
+  fi
+  if [ "$v1count" -eq 0 ]; then
+    # No v1 blobs at all — just stamp the marker (dotenv/file-only store).
+    printf '2\n' > "$SECRETS_DIR/$STORE_FORMAT_FILE_NAME"
+    ensure_store_protections
+    git -C "$SECRETS_DIR" add -A
+    git -C "$SECRETS_DIR" commit -m "migrate: finalize store format v2" >/dev/null 2>&1 || true
+    git -C "$SECRETS_DIR" remote get-url origin >/dev/null 2>&1 && git -C "$SECRETS_DIR" push >/dev/null 2>&1 || true
+    info "Store finalized to format v2 (no v1 blobs to drop)."
+    return 0
+  fi
+
+  # (gate 3) operator confirms every machine is upgraded.
+  if [ "$force" != true ]; then
+    echo "Finalize will drop $v1count v1 blob(s) and stamp the store format v2."
+    echo "Any machine still running a pre-v2 'secrets' will stop seeing 'properties'"
+    echo "externals from this store until it upgrades (git pull in the secrets repo)."
+    printf "Confirm every machine is upgraded? Type 'yes': "
+    local reply=""
+    read -r reply < /dev/tty 2>/dev/null || read -r reply || true
+    [ "$reply" = "yes" ] || die "Finalize aborted — no confirmation."
+  fi
+
+  # Recovery tag BEFORE any mutation: points at the pre-finalize commit (v1
+  # blobs intact, no marker), so `git checkout <tag>` fully restores v1.
+  local tag="pre-v2-migrate-$(git -C "$SECRETS_DIR" rev-parse --short HEAD 2>/dev/null || echo unknown)"
+  git -C "$SECRETS_DIR" tag "$tag" >/dev/null 2>&1 || true
+
+  # Stamp the marker FIRST, then drop v1 blobs. If finalize crashes between
+  # the two, the store reads as v2 and the (verified) v2 twins serve every
+  # upgraded client; any leftover v1 blobs are harmless orphans (referenced by
+  # no v2 suffix lookup) that `secrets verify` will flag and the operator can
+  # remove. A re-run of --finalize early-returns (store is already v2).
+  printf '2\n' > "$SECRETS_DIR/$STORE_FORMAT_FILE_NAME"
+  while IFS= read -r f; do
+    [ -f "$f" ] || continue
+    rm -f "$f"
+  done < <(find "$SECRETS_DIR" -type f -name '*.gradle-properties.age')
+
+  ensure_store_protections
+  git -C "$SECRETS_DIR" add -A
+  git -C "$SECRETS_DIR" commit -m "migrate: finalize store format v2 (drop $v1count v1 blob(s))" >/dev/null 2>&1 || true
+  git -C "$SECRETS_DIR" remote get-url origin >/dev/null 2>&1 && git -C "$SECRETS_DIR" push >/dev/null 2>&1 || true
+  info "Store finalized to format v2. Dropped $v1count v1 blob(s). Recovery tag in the store: $tag"
+}
+
+cmd_migrate() {
+  resolve_store
+  check_initialized
+  local dry_run=false finalize=false force=false
+  while [ $# -gt 0 ]; do
+    case "$1" in
+      --dry-run)     dry_run=true; shift ;;
+      --finalize)    finalize=true; shift ;;
+      --yes)         force=true; shift ;;
+      -*)            die "Unknown migrate flag: $1. Usage: secrets migrate [--dry-run | --finalize] [--yes]" ;;
+      *)             die "migrate takes no project argument. Run it from inside a project (copy-forward) or use --finalize (store-wide)." ;;
+    esac
+  done
+  if [ "$finalize" = true ]; then
+    _migrate_finalize "$force"
+  else
+    _migrate_project "$dry_run"
+  fi
+}
+
 cmd_help() {
   cat << 'EOF'
 secrets — encrypted secret file sync between machines
@@ -2087,6 +2302,8 @@ Usage:
   secrets rekey                 Re-encrypt all secrets with a new key
   secrets verify [project]      Check the manifest against the store + decrypt every blob
   secrets verify --all          Decrypt-test every blob in every project (integrity gate)
+  secrets migrate [--dry-run]   Copy-forward this project's blobs to store format v2
+  secrets migrate --finalize    Drop v1 blobs and mark the store v2 (after verify)
   secrets which                 Show the active store, manifest, and external entries
   secrets where                 Alias for `which`
   secrets status                Alias for `which`
@@ -2256,6 +2473,7 @@ case "${1:-help}" in
   rm)     cmd_rm "${2:-}" ;;
   rekey)  cmd_rekey ;;
   verify) shift; cmd_verify "$@" ;;
+  migrate) shift; cmd_migrate "$@" ;;
   which|where|status) cmd_which ;;
   help|--help|-h) cmd_help ;;
   *)      die "Unknown command: $1. Run 'secrets help' for usage." ;;

test/manifest.bats

@@ -269,7 +269,7 @@ m_file_src()   { mkdir -p "$HOME/keystores"; printf 'KS\x00\x01\x02\xffDATA\n' >
   run jq -r '.external[] | select(.type == "file") | .path' .secrets.json
   [ "$output" = "~/keystores/upload.keystore" ]
   # stage 1: blob naming stays legacy-compatible
-  run bash -c "ls $SECRETS_DIR/absorbproj/external/*.gradle-properties.age"
+  run bash -c "ls $SECRETS_DIR/absorbproj/external/*.properties.age"
   [ "$status" -eq 0 ]
 }
 
@@ -296,7 +296,7 @@ m_file_src()   { mkdir -p "$HOME/keystores"; printf 'KS\x00\x01\x02\xffDATA\n' >
   run "$SECRETS_BIN" push jsonextproj
   [ "$status" -eq 0 ]
   [[ "$output" == *"Extracted 1 key"* ]] || false
-  run bash -c "ls $SECRETS_DIR/jsonextproj/external/*.gradle-properties.age"
+  run bash -c "ls $SECRETS_DIR/jsonextproj/external/*.properties.age"
   [ "$status" -eq 0 ]
 }
 

test/migrate.bats

@@ -0,0 +1,322 @@
+#!/usr/bin/env bats
+# EGB-703 store-format-v2: marker, format-aware suffix, migrate (dry-run /
+# copy-forward / finalize). bash 3.2: every standalone [[ ]] ends with || false.
+
+load test_helper
+
+# A v1 (legacy) store: born-v2 init, then strip the marker so it reads as v1
+# and pushes write the legacy .gradle-properties.age suffix.
+make_v1_store() {
+  init_with_remote
+  rm -f "$SECRETS_DIR/.secrets-format"
+}
+m_gradle_src() { mkdir -p "$HOME/.gradle"; printf '%s' "$1" > "$HOME/.gradle/gradle.properties"; }
+m_file_src()   { mkdir -p "$HOME/keystores"; printf 'KS\x00\x01\x02\xffDATA\n' > "$HOME/keystores/upload.keystore"; }
+
+# ─── Format marker + format-aware suffix (increment 1) ────────────────
+
+@test "init stamps the store format marker as v2 (born-v2)" {
+  init_with_remote
+  [ -f "$SECRETS_DIR/.secrets-format" ]
+  [ "$(cat "$SECRETS_DIR/.secrets-format")" = "2" ]
+}
+
+@test "which prints format v2 for a born-v2 store" {
+  init_with_remote
+  create_project_dir whichv2
+  run "$SECRETS_BIN" which
+  [ "$status" -eq 0 ]
+  [[ "$output" == *"format: v2"* ]] || false
+}
+
+@test "which prints format v1 for a markerless (legacy) store" {
+  make_v1_store
+  create_project_dir whichv1
+  run "$SECRETS_BIN" which
+  [ "$status" -eq 0 ]
+  [[ "$output" == *"format: v1"* ]] || false
+}
+
+@test "push on a born-v2 store writes the properties blob as .properties.age" {
+  init_with_remote
+  m_gradle_src $'beaconClerkPkTest=pk_test_abc\n'
+  create_project_dir v2push
+  printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files
+  "$SECRETS_BIN" push v2push >/dev/null 2>&1
+  run bash -c "ls $SECRETS_DIR/v2push/external/*.properties.age"
+  [ "$status" -eq 0 ]
+  run bash -c "ls $SECRETS_DIR/v2push/external/*.gradle-properties.age 2>/dev/null"
+  [ "$status" -ne 0 ]
+}
+
+@test "push on a v1 store still writes .gradle-properties.age (back-compat)" {
+  make_v1_store
+  m_gradle_src $'beaconClerkPkTest=pk_test_abc\n'
+  create_project_dir v1push
+  printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files
+  "$SECRETS_BIN" push v1push >/dev/null 2>&1
+  run bash -c "ls $SECRETS_DIR/v1push/external/*.gradle-properties.age"
+  [ "$status" -eq 0 ]
+}
+
+@test "the format marker is committed, not gitignored" {
+  init_with_remote
+  create_project_dir markercommit
+  "$SECRETS_BIN" push markercommit >/dev/null 2>&1
+  run bash -c "git -C $SECRETS_DIR ls-files | grep -qx .secrets-format"
+  [ "$status" -eq 0 ]
+}
+
+# ─── migrate --dry-run / copy-forward (increment 2) ───────────────────
+
+@test "migrate --dry-run reports the rename and writes nothing" {
+  make_v1_store
+  m_gradle_src $'beaconClerkPkTest=pk_test_abc\n'
+  create_project_dir dryproj
+  printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files
+  "$SECRETS_BIN" push dryproj >/dev/null 2>&1
+  run "$SECRETS_BIN" migrate --dry-run
+  [ "$status" -eq 0 ]
+  [[ "$output" == *"would migrate"* ]] || false
+  # nothing written
+  run bash -c "ls $SECRETS_DIR/dryproj/external/*.properties.age 2>/dev/null"
+  [ "$status" -ne 0 ]
+  # marker still absent (store still v1)
+  [ ! -f "$SECRETS_DIR/.secrets-format" ]
+}
+
+@test "migrate --dry-run on a dotenv-only project reports nothing to migrate" {
+  make_v1_store
+  create_project_dir dotenvonly
+  "$SECRETS_BIN" push dotenvonly >/dev/null 2>&1
+  run "$SECRETS_BIN" migrate --dry-run
+  [ "$status" -eq 0 ]
+  [[ "$output" == *"0 blob(s) would be copy-forwarded"* ]] || false
+}
+
+@test "migrate copy-forward creates the v2 twin and keeps the v1 blob (byte-identical)" {
+  make_v1_store
+  m_gradle_src $'beaconClerkPkTest=pk_test_abc\n'
+  create_project_dir cfproj
+  printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files
+  "$SECRETS_BIN" push cfproj >/dev/null 2>&1
+  local old; old=$(ls "$SECRETS_DIR/cfproj/external/"*.gradle-properties.age)
+  run "$SECRETS_BIN" migrate
+  [ "$status" -eq 0 ]
+  local new; new=$(ls "$SECRETS_DIR/cfproj/external/"*.properties.age)
+  [ -f "$old" ]   # v1 kept (non-destructive)
+  [ -f "$new" ]   # v2 twin written
+  cmp -s "$old" "$new"   # byte-identical ciphertext copy
+}
+
+@test "migrate copy-forward is idempotent" {
+  make_v1_store
+  m_gradle_src $'beaconClerkPkTest=pk_test_abc\n'
+  create_project_dir idemproj
+  printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files
+  "$SECRETS_BIN" push idemproj >/dev/null 2>&1
+  "$SECRETS_BIN" migrate >/dev/null 2>&1
+  run "$SECRETS_BIN" migrate
+  [ "$status" -eq 0 ]
+  [[ "$output" == *"1 already present"* ]] || false
+  run bash -c "ls $SECRETS_DIR/idemproj/external/*.properties.age | wc -l | tr -d ' '"
+  [ "$output" = "1" ]
+}
+
+@test "migrate with no manifest in cwd dies with a directed message" {
+  make_v1_store
+  local dir="$WORK_DIR/nomanifest"; mkdir -p "$dir"; cd "$dir"
+  run "$SECRETS_BIN" migrate
+  [ "$status" -eq 1 ]
+  [[ "$output" == *".secrets.json"* ]] || false
+}
+
+@test "migrate on an already-v2 store is a no-op" {
+  init_with_remote
+  create_project_dir alreadyv2
+  run "$SECRETS_BIN" migrate
+  [ "$status" -eq 0 ]
+  [[ "$output" == *"already format v2"* ]] || false
+}
+
+@test "migrate unknown flag dies with usage" {
+  init_with_remote
+  create_project_dir mgflag
+  run "$SECRETS_BIN" migrate --bogus
+  [ "$status" -eq 1 ]
+  [[ "$output" == *"Unknown migrate flag"* ]] || false
+}
+
+@test "migrate leaves dotenv and file blobs untouched" {
+  make_v1_store
+  m_gradle_src $'beaconClerkPkTest=pk_test_abc\n'
+  m_file_src
+  create_project_dir mixproj
+  printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\nfile ~/keystores/upload.keystore\n' > .secrets-files
+  "$SECRETS_BIN" push mixproj >/dev/null 2>&1
+  local envblob; envblob=$(ls "$SECRETS_DIR/mixproj/".env.age)
+  local fileblob; fileblob=$(ls "$SECRETS_DIR/mixproj/external/"*.file.age)
+  local envsum; envsum=$(cksum "$envblob")
+  local filesum; filesum=$(cksum "$fileblob")
+  "$SECRETS_BIN" migrate >/dev/null 2>&1
+  [ "$(cksum "$envblob")" = "$envsum" ]    # dotenv blob unchanged
+  [ "$(cksum "$fileblob")" = "$filesum" ]  # file blob unchanged
+}
+
+# ─── migrate --finalize (increment 3) ─────────────────────────────────
+
+@test "finalize refuses when verify --all is not green" {
+  make_v1_store
+  m_gradle_src $'beaconClerkPkTest=pk_test_abc\n'
+  create_project_dir failverify
+  printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files
+  "$SECRETS_BIN" push failverify >/dev/null 2>&1
+  "$SECRETS_BIN" migrate >/dev/null 2>&1
+  # corrupt the v2 twin so verify --all fails
+  printf 'garbage' > "$SECRETS_DIR/failverify/external/"*.properties.age
+  run "$SECRETS_BIN" migrate --finalize --yes
+  [ "$status" -eq 1 ]
+  [[ "$output" == *"not green"* ]] || false
+  # marker not stamped; v1 blob still present
+  [ ! -f "$SECRETS_DIR/.secrets-format" ]
+  run bash -c "ls $SECRETS_DIR/failverify/external/*.gradle-properties.age"
+  [ "$status" -eq 0 ]
+}
+
+@test "finalize refuses an un-twinned v1 blob (project not migrated)" {
+  make_v1_store
+  m_gradle_src $'beaconClerkPkTest=pk_test_abc\n'
+  create_project_dir untwinned
+  printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files
+  "$SECRETS_BIN" push untwinned >/dev/null 2>&1
+  # do NOT migrate — leave the v1 blob with no twin
+  run "$SECRETS_BIN" migrate --finalize --yes
+  [ "$status" -eq 1 ]
+  [[ "$output" == *"no v2 twin"* ]] || false
+  run bash -c "ls $SECRETS_DIR/untwinned/external/*.gradle-properties.age"
+  [ "$status" -eq 0 ]
+}
+
+@test "finalize green path drops v1, keeps v2, stamps the marker" {
+  make_v1_store
+  m_gradle_src $'beaconClerkPkTest=pk_test_abc\n'
+  create_project_dir finproj
+  printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files
+  "$SECRETS_BIN" push finproj >/dev/null 2>&1
+  "$SECRETS_BIN" migrate >/dev/null 2>&1
+  run "$SECRETS_BIN" migrate --finalize --yes
+  [ "$status" -eq 0 ]
+  [ "$(cat "$SECRETS_DIR/.secrets-format")" = "2" ]
+  run bash -c "ls $SECRETS_DIR/finproj/external/*.properties.age"
+  [ "$status" -eq 0 ]
+  run bash -c "ls $SECRETS_DIR/finproj/external/*.gradle-properties.age 2>/dev/null"
+  [ "$status" -ne 0 ]
+}
+
+@test "finalize cuts a recovery tag before deleting v1 blobs" {
+  make_v1_store
+  m_gradle_src $'beaconClerkPkTest=pk_test_abc\n'
+  create_project_dir tagproj
+  printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files
+  "$SECRETS_BIN" push tagproj >/dev/null 2>&1
+  "$SECRETS_BIN" migrate >/dev/null 2>&1
+  "$SECRETS_BIN" migrate --finalize --yes >/dev/null 2>&1
+  local tag; tag=$(git -C "$SECRETS_DIR" tag | grep '^pre-v2-migrate-')
+  [ -n "$tag" ]
+  # the tagged commit still contains the v1 blob (tag cut before delete)
+  run bash -c "git -C $SECRETS_DIR ls-tree -r --name-only $tag | grep -q gradle-properties.age"
+  [ "$status" -eq 0 ]
+}
+
+@test "finalize without --yes aborts when not confirmed" {
+  make_v1_store
+  m_gradle_src $'beaconClerkPkTest=pk_test_abc\n'
+  create_project_dir confproj
+  printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files
+  "$SECRETS_BIN" push confproj >/dev/null 2>&1
+  "$SECRETS_BIN" migrate >/dev/null 2>&1
+  run bash -c "echo '' | $SECRETS_BIN migrate --finalize"
+  [ "$status" -eq 1 ]
+  [[ "$output" == *"aborted"* ]] || false
+  [ ! -f "$SECRETS_DIR/.secrets-format" ]
+}
+
+@test "verify is green during the migration window (v2 twin is not a spurious orphan)" {
+  # Regression: after copy-forward the store is still v1, so verify computed the
+  # external blob path as .gradle-properties.age and flagged the .properties.age
+  # twin as an orphan, failing verify mid-migration. The orphan set now accounts
+  # for both suffix forms.
+  make_v1_store
+  m_gradle_src $'beaconClerkPkTest=pk_test_abc\n'
+  create_project_dir windowverify
+  printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files
+  "$SECRETS_BIN" push windowverify >/dev/null 2>&1
+  "$SECRETS_BIN" migrate >/dev/null 2>&1
+  run "$SECRETS_BIN" verify windowverify
+  [ "$status" -eq 0 ]
+}
+
+@test "v1 client still reads during the migration window (after copy-forward, before finalize)" {
+  make_v1_store
+  m_gradle_src $'beaconClerkPkTest=pk_test_abc\n'
+  create_project_dir windowproj
+  printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files
+  "$SECRETS_BIN" push windowproj >/dev/null 2>&1
+  "$SECRETS_BIN" migrate >/dev/null 2>&1
+  # store is still v1 (markerless); pull uses the .gradle-properties.age blob
+  rm "$HOME/.gradle/gradle.properties"
+  run "$SECRETS_BIN" pull windowproj
+  [ "$status" -eq 0 ]
+  grep -q '^beaconClerkPkTest=pk_test_abc$' "$HOME/.gradle/gradle.properties"
+}
+
+@test "post-finalize pull reads the v2 blob" {
+  make_v1_store
+  m_gradle_src $'beaconClerkPkTest=pk_test_abc\n'
+  create_project_dir postfin
+  printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files
+  "$SECRETS_BIN" push postfin >/dev/null 2>&1
+  "$SECRETS_BIN" migrate >/dev/null 2>&1
+  "$SECRETS_BIN" migrate --finalize --yes >/dev/null 2>&1
+  rm "$HOME/.gradle/gradle.properties"
+  run "$SECRETS_BIN" pull postfin
+  [ "$status" -eq 0 ]
+  grep -q '^beaconClerkPkTest=pk_test_abc$' "$HOME/.gradle/gradle.properties"
+}
+
+@test "finalize on a v1 dotenv-only store stamps the marker (no v1 blobs to drop)" {
+  make_v1_store
+  create_project_dir dotenvfin
+  "$SECRETS_BIN" push dotenvfin >/dev/null 2>&1
+  # no gradle-properties externals → v1count==0 path; no confirmation needed
+  run "$SECRETS_BIN" migrate --finalize
+  [ "$status" -eq 0 ]
+  [[ "$output" == *"no v1 blobs"* ]] || false
+  [ "$(cat "$SECRETS_DIR/.secrets-format")" = "2" ]
+}
+
+@test "migrate with a positional argument dies" {
+  init_with_remote
+  create_project_dir mgpos
+  run "$SECRETS_BIN" migrate someproject
+  [ "$status" -eq 1 ]
+  [[ "$output" == *"no project argument"* ]] || false
+}
+
+@test "finalize on an already-v2 store is a no-op" {
+  init_with_remote
+  create_project_dir finv2
+  run "$SECRETS_BIN" migrate --finalize --yes
+  [ "$status" -eq 0 ]
+  [[ "$output" == *"already format v2"* ]] || false
+}
+
+@test "_store_format reads a garbage marker as v1 (strict parse)" {
+  make_v1_store
+  create_project_dir garbagemarker
+  # a non-"2" marker (e.g. a truncated/garbled value) must read as v1, not v2
+  printf 'v2-ish-garbage\n' > "$SECRETS_DIR/.secrets-format"
+  run "$SECRETS_BIN" which
+  [ "$status" -eq 0 ]
+  [[ "$output" == *"format: v1"* ]] || false
+}

test/secrets.bats

@@ -1081,7 +1081,7 @@ gradle_project() {
   run "$SECRETS_BIN" push gproj
   [ "$status" -eq 0 ]
   [[ "$output" == *"Extracted 2 key"* ]] || false
-  run bash -c "ls $SECRETS_DIR/gproj/external/*.gradle-properties.age"
+  run bash -c "ls $SECRETS_DIR/gproj/external/*.properties.age"
   [ "$status" -eq 0 ]
 }
 
@@ -1368,7 +1368,7 @@ gradle_project() {
   run "$SECRETS_BIN" push -w
   [ "$status" -eq 0 ]
   # External blob pushed exactly once (not once per workspace)
-  run bash -c "ls $SECRETS_DIR/mono/external/*.gradle-properties.age 2>/dev/null | wc -l | tr -d ' '"
+  run bash -c "ls $SECRETS_DIR/mono/external/*.properties.age 2>/dev/null | wc -l | tr -d ' '"
   [ "$output" = "1" ]
 }