Merge pull request 'v0.5.0.0 feat: secrets verify integrity check (EGB-698)' (#4) from brian/egb-698-secrets-verify-command into main

CHANGELOG.md

@@ -5,6 +5,23 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to a four-digit MAJOR.MINOR.PATCH.MICRO version scheme.
 
+## [0.5.0.0] - 2026-06-07
+
+### Added
+
+- **`secrets verify` (EGB-698)** — a read-only integrity check. Run it in a
+  project to cross-check the committed `.secrets.json` against the store both
+  ways (entries declared but missing from the store, and stored blobs with no
+  manifest entry) and decrypt-test every blob with your current key. Catches a
+  partially-synced store, a stale key, or a manifest that has drifted from the
+  store. Plaintext is streamed to `/dev/null` and never written to disk.
+- **`secrets verify --all`** — decrypt-tests every blob in every project in the
+  store: a fast store-wide integrity sweep. (The store carries no manifests, so
+  `--all` checks decryptability only, not manifest consistency.)
+- Both modes recurse the whole project tree, so nested entries and external
+  files are covered. `secrets verify` exits non-zero on any problem, so it can
+  gate CI or a future store migration.
+
 ## [0.4.0.0] - 2026-06-07
 
 ### Added

CLAUDE.md

@@ -56,12 +56,13 @@ skips security specialist + red team, and Step 11 skips adversarial review.
 
 ## Architecture
 
-Single bash script (`secrets`) with subcommands: init, push, pull, list, rm, rekey.
+Single bash script (`secrets`) with subcommands: init, push, pull, list, rm, rekey, verify.
 
 - Encryption: `age` with key files (not passphrases — age passphrases are non-scriptable)
 - Storage: Private git repo at `~/.secrets/`
 - Convention: Tracks `.env`, `.env.*`, and `.dev.vars` (not `.envrc`, `.environment-*`)
 - Manifest (EGB-677 stage 1): committed `.secrets.json` is the source of truth for what syncs — `dotenv[]` (project-relative, nested ok, `@` allowed; rail rejects `..`/absolute/symlink) + `external[]` (`properties`/`file`). Push discovery auto-adds (gated by committed `options.autoAdd`, default ON; `--frozen`/`--dry-run` overrides), bootstraps the manifest on first push (written only after ≥1 blob encrypts), and absorbs a legacy `.secrets-files` (gradle-properties → `properties`; on pull the legacy file is superseded with a warning). v1 store layout unchanged in stage 1: nested entries land at `<project>/<relpath>.age`; `properties` blobs keep the legacy `.gradle-properties.age` suffix until the stage-2 store migration. jq is a hard dep only when a manifest exists/is written; manifest-less projects run jq-free (manifest features skipped with a notice). `check_cmd` prints platform-aware install hints.
+- Verify (EGB-698): `secrets verify` is a read-only integrity check. Default mode (current project) cross-checks `$PWD/.secrets.json` against `$SECRETS_DIR/<project>/` both ways (declared-but-missing blobs + orphaned blobs) and decrypt-tests every blob (dotenv + external) by streaming plaintext to `/dev/null` (never written to disk). `secrets verify --all` decrypt-tests every blob in every project (integrity only — the store carries no manifests, so consistency can't be checked store-wide). Both recurse the whole project tree (`find -type f`, same as rekey/list). Exits non-zero on any finding so it can gate the stage-2 `migrate --finalize` and CI. The store deliberately holds no manifest — `.secrets.json` is committed in each project's own repo and read from `$PWD`.
 - External files: `.secrets-files` manifest tracks designated keys from files outside the project (e.g. `~/.gradle/gradle.properties`, merged not overwritten — EGB-531) and whole binary files (type `file`, e.g. an Android upload keystore — EGB-652); see below
 - Workspaces: `--workspaces` flag reads `package.json` workspaces, requires `jq`
 - Safety: Pre-commit hook rejects plaintext secret files (`.env`, `.dev.vars`, `gradle.properties`)
@@ -74,7 +75,7 @@ secrets              # CLI script (~2000 lines bash)
 hooks/pre-commit     # Pre-commit hook template
 test/
   secrets.bats       # bats-core test suite (133 tests)
-  manifest.bats      # EGB-677 .secrets.json manifest tests (60 tests)
+  manifest.bats      # EGB-677 .secrets.json manifest tests (77 tests)
   test_helper.bash   # Shared setup/teardown
 README.md            # User-facing documentation
 CLAUDE.md            # This file

README.md

@@ -169,6 +169,8 @@ secrets clear
 | `secrets list` | Show all projects that have stored secrets |
 | `secrets rm <project>` | Delete a project's secrets from the store |
 | `secrets rekey` | Generate a new encryption key and re-encrypt everything |
+| `secrets verify [project]` | Check the current project's `.secrets.json` against the store (missing/orphaned blobs) and decrypt every blob. `[project]` overrides the store directory name; the manifest is still read from the current directory |
+| `secrets verify --all` | Decrypt-test every blob in every project — a store-wide integrity sweep |
 
 ### Automatic project detection
 
@@ -491,10 +493,12 @@ For complete rotation with no historical exposure, create a fresh `~/.secrets/`
 
 **"'jq' is not installed"** — Manifest features need `jq`. The error prints the install command for your platform. Manifest-less projects work without it.
 
+**Not sure the store is intact?** — Run `secrets verify` in a project to check its `.secrets.json` against the store (declared-but-missing blobs and orphaned blobs) and decrypt-test every blob with your current key. Use `secrets verify --all` for a store-wide decrypt sweep across every project. It's read-only — plaintext is streamed to `/dev/null`, never written to disk — and exits non-zero if anything is wrong, so it's safe to run in CI.
+
 ## Development
 
 ```bash
-# Run the test suite (193 tests across both files)
+# Run the test suite (210 tests across both files)
 brew install bats-core
 bats test/
 

VERSION

@@ -1 +1 @@
-0.4.0.0
+0.5.0.0

secrets

@@ -1907,6 +1907,165 @@ cmd_which() {
   fi
 }
 
+# Decrypt-test one blob with the current key. Plaintext is streamed to
+# /dev/null and never written to disk (read-only contract). Returns 0 if the
+# blob decrypts, non-zero otherwise.
+_verify_blob_decrypts() {
+  age -d -i "$KEY_FILE" "$1" >/dev/null 2>&1
+}
+
+# `secrets verify --all` — store-wide decrypt sweep. Decrypt-tests every blob
+# in every project. No manifest consistency check: the store carries only
+# ciphertext (manifests live in each project's repo), so orphan/missing
+# detection is impossible store-wide. This is the migration integrity gate.
+_verify_all() {
+  local checked=0 failed=0 dir project f rel
+  for dir in "$SECRETS_DIR"/*/; do
+    [ -d "$dir" ] || continue
+    project=$(basename "$dir")
+    [[ "$project" == .* ]] && continue
+    # Recurse the whole project tree (top-level / nested / external) — never a
+    # non-recursive glob, or nested blobs would be silently skipped (the exact
+    # class of bug that orphaned nested blobs on rekey before v0.4.0.0).
+    while IFS= read -r f; do
+      [ -f "$f" ] || continue
+      checked=$((checked + 1))
+      if ! _verify_blob_decrypts "$f"; then
+        rel="${f#"$SECRETS_DIR"/}"
+        echo "FAIL: $rel does not decrypt with the current key." >&2
+        failed=$((failed + 1))
+      fi
+    done < <(find "$dir" -type f -name '*.age')
+  done
+  if [ "$checked" -eq 0 ]; then
+    info "verify --all: store is empty — nothing to check."
+    return 0
+  fi
+  if [ "$failed" -gt 0 ]; then
+    echo "verify --all: $failed of $checked blob(s) failed to decrypt." >&2
+    return 1
+  fi
+  info "verify --all: OK — all $checked blob(s) decrypt with the current key (integrity only; run 'secrets verify' in a project for manifest consistency)."
+}
+
+# `secrets verify [project]` — current-project consistency + decrypt check.
+# Cross-checks the committed .secrets.json against the store both ways
+# (declared-but-missing blobs, orphaned blobs) and decrypt-tests every blob.
+_verify_project() {
+  local explicit_project="$1"
+  local manifest="$PWD/$SECRETS_JSON_NAME"
+  if [ ! -e "$manifest" ]; then
+    die "No $SECRETS_JSON_NAME in $PWD.
+  'secrets verify' checks a project's manifest against the store. Either cd
+  into a project that has a $SECRETS_JSON_NAME, or run 'secrets verify --all'
+  for a store-wide decrypt sweep."
+  fi
+  _check_manifest_file "$manifest"   # fatal on symlink / malformed / version
+
+  local project
+  project=$(derive_project_name "$explicit_project")
+  local pdir="$SECRETS_DIR/$project"
+
+  local findings=0 checked=0
+  # `expected` accumulates the store-relative blob paths the manifest implies,
+  # newline-framed (leading + trailing \n per entry) so the orphan walk can
+  # test membership. bash 3.2 has no associative arrays — this string-set +
+  # case match mirrors the `seen` pattern in _external_entries_for_push.
+  local expected=$'\n'
+
+  # ── dotenv entries: rail + missing-blob + decrypt ──
+  local rel blob
+  while IFS= read -r rel; do
+    [ -n "$rel" ] || continue
+    if ! _validate_dotenv_rel_path "$rel" 2>/dev/null; then
+      echo "FINDING: unsafe dotenv path in $SECRETS_JSON_NAME: '$rel' (will be refused)." >&2
+      findings=$((findings + 1))
+      # Still account for it so a matching stray blob isn't ALSO flagged as an
+      # orphan (one bad entry → one finding, not two).
+      expected="$expected$rel.age"$'\n'
+      continue
+    fi
+    expected="$expected$rel.age"$'\n'
+    blob="$pdir/$rel.age"
+    if [ ! -f "$blob" ]; then
+      echo "FINDING: '$rel' is declared in $SECRETS_JSON_NAME but has no blob in the store ($project/$rel.age missing). Run 'secrets push'." >&2
+      findings=$((findings + 1))
+      continue
+    fi
+    checked=$((checked + 1))
+    if ! _verify_blob_decrypts "$blob"; then
+      echo "FINDING: blob for '$rel' ($project/$rel.age) does not decrypt with the current key." >&2
+      findings=$((findings + 1))
+    fi
+  done < <(jq -r '.dotenv // [] | .[]' "$manifest")
+
+  # ── external entries: missing-blob + decrypt ──
+  # verify only needs type + path to locate the blob; keys are irrelevant here.
+  local etype epath slug erel eblob
+  while IFS=$'\t' read -r etype epath _; do
+    [ -n "$etype" ] || continue
+    slug=$(_secrets_files_slug "$epath")
+    erel="external/$slug.$etype.age"
+    expected="$expected$erel"$'\n'
+    eblob="$pdir/$erel"
+    if [ ! -f "$eblob" ]; then
+      echo "FINDING: external '$epath' ($etype) is declared but has no blob in the store ($project/$erel missing). Run 'secrets push'." >&2
+      findings=$((findings + 1))
+      continue
+    fi
+    checked=$((checked + 1))
+    if ! _verify_blob_decrypts "$eblob"; then
+      echo "FINDING: external blob for '$epath' ($project/$erel) does not decrypt with the current key." >&2
+      findings=$((findings + 1))
+    fi
+  done < <(_json_external_entries "$manifest")
+
+  # ── orphan detection: any stored blob the manifest doesn't account for ──
+  if [ -d "$pdir" ]; then
+    local f frel
+    while IFS= read -r f; do
+      [ -f "$f" ] || continue
+      frel="${f#"$pdir"/}"
+      case "$expected" in
+        *$'\n'"$frel"$'\n'*) : ;;   # accounted for by a manifest entry
+        *)
+          echo "FINDING: orphan blob $project/$frel has no entry in $SECRETS_JSON_NAME." >&2
+          findings=$((findings + 1))
+          ;;
+      esac
+    done < <(find "$pdir" -type f -name '*.age')
+  fi
+
+  if [ "$findings" -gt 0 ]; then
+    echo "verify: $findings finding(s) for project '$project'." >&2
+    return 1
+  fi
+  info "verify: OK — $checked blob(s) verified for '$project' (manifest and store agree; all decrypt)."
+}
+
+# `secrets verify [--all] [project]` — read-only integrity check. Exits
+# non-zero on any finding so it can gate `migrate --finalize` and CI.
+cmd_verify() {
+  resolve_store
+  check_initialized
+  check_key
+
+  local all=false explicit_project=""
+  while [ $# -gt 0 ]; do
+    case "$1" in
+      --all) all=true; shift ;;
+      -*)    die "Unknown verify flag: $1. Usage: secrets verify [--all] [project]" ;;
+      *)     explicit_project="$1"; shift ;;
+    esac
+  done
+
+  if [ "$all" = true ]; then
+    _verify_all
+  else
+    _verify_project "$explicit_project"
+  fi
+}
+
 cmd_help() {
   cat << 'EOF'
 secrets — encrypted secret file sync between machines
@@ -1926,6 +2085,8 @@ Usage:
   secrets list                  List all projects and their secret files
   secrets rm <project>          Remove a project's secrets from the repo
   secrets rekey                 Re-encrypt all secrets with a new key
+  secrets verify [project]      Check the manifest against the store + decrypt every blob
+  secrets verify --all          Decrypt-test every blob in every project (integrity gate)
   secrets which                 Show the active store, manifest, and external entries
   secrets where                 Alias for `which`
   secrets status                Alias for `which`
@@ -2094,6 +2255,7 @@ case "${1:-help}" in
   list)   cmd_list ;;
   rm)     cmd_rm "${2:-}" ;;
   rekey)  cmd_rekey ;;
+  verify) shift; cmd_verify "$@" ;;
   which|where|status) cmd_which ;;
   help|--help|-h) cmd_help ;;
   *)      die "Unknown command: $1. Run 'secrets help' for usage." ;;

test/manifest.bats

@@ -731,3 +731,186 @@ m_nojq_path() {
   [ "$status" -eq 0 ]
   [[ "$output" == *"packages/web/.env.development"* ]] || false
 }
+
+# ─── K: secrets verify — manifest↔store consistency + decrypt integrity (EGB-698) ─
+
+@test "verify: clean pushed project reports OK and exits 0" {
+  init_with_remote
+  create_project_dir verifyok
+  "$SECRETS_BIN" push >/dev/null 2>&1
+  run "$SECRETS_BIN" verify
+  [ "$status" -eq 0 ]
+  [[ "$output" == *"OK"* ]] || false
+}
+
+@test "verify: nested + external entries all pass" {
+  init_with_remote
+  m_gradle_src $'beaconClerkPkTest=pk_test_abc\n'
+  create_project_dir verifymix
+  mkdir -p packages/web
+  echo "N=nested" > packages/web/.env.development
+  "$SECRETS_BIN" add packages/web/.env.development >/dev/null
+  printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files
+  "$SECRETS_BIN" push >/dev/null 2>&1
+  run "$SECRETS_BIN" verify
+  [ "$status" -eq 0 ]
+  # Pin the count so a silent under-count (e.g. skipping the nested or external
+  # blob while still exiting 0) is caught: 3 dotenv + 1 external = 4.
+  [[ "$output" == *"4 blob(s) verified"* ]] || false
+}
+
+@test "verify: declared-but-missing blob is a finding (exit 1)" {
+  init_with_remote
+  create_project_dir verifymiss
+  "$SECRETS_BIN" push >/dev/null 2>&1
+  rm "$SECRETS_DIR/verifymiss/.env.staging.age"
+  run "$SECRETS_BIN" verify
+  [ "$status" -eq 1 ]
+  [[ "$output" == *".env.staging"* ]] || false
+}
+
+@test "verify: a blob that fails to decrypt is a finding (exit 1)" {
+  init_with_remote
+  create_project_dir verifycorrupt
+  "$SECRETS_BIN" push >/dev/null 2>&1
+  printf 'not-a-valid-age-blob' > "$SECRETS_DIR/verifycorrupt/.env.age"
+  run "$SECRETS_BIN" verify
+  [ "$status" -eq 1 ]
+  [[ "$output" == *".env"* ]] || false
+}
+
+@test "verify: orphan blob (no manifest entry) is a finding (exit 1)" {
+  init_with_remote
+  create_project_dir verifyorphan
+  "$SECRETS_BIN" push >/dev/null 2>&1
+  # A valid, decryptable blob with no manifest entry — pure consistency miss.
+  cp "$SECRETS_DIR/verifyorphan/.env.age" "$SECRETS_DIR/verifyorphan/.stray.age"
+  run "$SECRETS_BIN" verify
+  [ "$status" -eq 1 ]
+  [[ "$output" == *"stray"* ]] || false
+}
+
+@test "verify: missing external blob is a finding (exit 1)" {
+  init_with_remote
+  m_gradle_src $'beaconClerkPkTest=pk_test_abc\n'
+  create_project_dir verifyextmiss
+  printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files
+  "$SECRETS_BIN" push >/dev/null 2>&1
+  rm "$SECRETS_DIR/verifyextmiss/external/"*.age
+  run "$SECRETS_BIN" verify
+  [ "$status" -eq 1 ]
+}
+
+@test "verify: no manifest in cwd dies with a directed message" {
+  init_with_remote
+  local dir="$WORK_DIR/verifynomani"; mkdir -p "$dir"; cd "$dir"
+  run "$SECRETS_BIN" verify
+  [ "$status" -eq 1 ]
+  [[ "$output" == *".secrets.json"* ]] || false
+  [[ "$output" == *"--all"* ]] || false
+}
+
+@test "verify: symlinked manifest is refused" {
+  init_with_remote
+  create_project_dir verifysymlink
+  "$SECRETS_BIN" push >/dev/null 2>&1
+  rm .secrets.json
+  ln -s /etc/hosts .secrets.json
+  run "$SECRETS_BIN" verify
+  [ "$status" -eq 1 ]
+  [[ "$output" == *"symlink"* ]] || false
+}
+
+@test "verify --all: clean store passes (decrypt-only)" {
+  init_with_remote
+  create_project_dir verifyall1
+  "$SECRETS_BIN" push >/dev/null 2>&1
+  cd "$WORK_DIR"
+  run "$SECRETS_BIN" verify --all
+  [ "$status" -eq 0 ]
+}
+
+@test "verify --all: a corrupted blob anywhere fails (exit 1)" {
+  init_with_remote
+  create_project_dir verifyall2
+  "$SECRETS_BIN" push >/dev/null 2>&1
+  printf 'garbage' > "$SECRETS_DIR/verifyall2/.env.age"
+  cd "$WORK_DIR"
+  run "$SECRETS_BIN" verify --all
+  [ "$status" -eq 1 ]
+}
+
+@test "verify --all: an orphan that decrypts is NOT flagged (no consistency check)" {
+  init_with_remote
+  create_project_dir verifyall3
+  "$SECRETS_BIN" push >/dev/null 2>&1
+  # Orphan blob that decrypts fine — default mode flags it, --all does not.
+  cp "$SECRETS_DIR/verifyall3/.env.age" "$SECRETS_DIR/verifyall3/.stray.age"
+  cd "$WORK_DIR"
+  run "$SECRETS_BIN" verify --all
+  [ "$status" -eq 0 ]
+}
+
+@test "verify: nested blob that fails to decrypt is caught (rekey-orphan guard)" {
+  init_with_remote
+  create_project_dir verifynested
+  mkdir -p packages/web
+  echo "N=nested" > packages/web/.env.development
+  "$SECRETS_BIN" add packages/web/.env.development >/dev/null
+  "$SECRETS_BIN" push >/dev/null 2>&1
+  printf 'broken' > "$SECRETS_DIR/verifynested/packages/web/.env.development.age"
+  run "$SECRETS_BIN" verify
+  [ "$status" -eq 1 ]
+  [[ "$output" == *"packages/web/.env.development"* ]] || false
+}
+
+@test "verify: external blob that fails to decrypt is a finding (exit 1)" {
+  init_with_remote
+  m_gradle_src $'beaconClerkPkTest=pk_test_abc\n'
+  create_project_dir verifyextcorrupt
+  printf 'gradle-properties ~/.gradle/gradle.properties beaconClerkPkTest\n' > .secrets-files
+  "$SECRETS_BIN" push >/dev/null 2>&1
+  printf 'garbage' > "$SECRETS_DIR/verifyextcorrupt/external/"*.age
+  run "$SECRETS_BIN" verify
+  [ "$status" -eq 1 ]
+  # Pin the decrypt-fail branch specifically, not just any external finding.
+  [[ "$output" == *"does not decrypt"* ]] || false
+}
+
+@test "verify: an unsafe dotenv path in the manifest is a finding (exit 1)" {
+  init_with_remote
+  create_project_dir verifyunsafe
+  "$SECRETS_BIN" push >/dev/null 2>&1
+  # Hand-edit the committed manifest to declare a traversal path the rail refuses.
+  jq '.dotenv += ["../evil"]' .secrets.json > .secrets.json.tmp && mv .secrets.json.tmp .secrets.json
+  run "$SECRETS_BIN" verify
+  [ "$status" -eq 1 ]
+  [[ "$output" == *"unsafe"* ]] || false
+}
+
+@test "verify: unknown flag dies with usage" {
+  init_with_remote
+  create_project_dir verifyflag
+  "$SECRETS_BIN" push >/dev/null 2>&1
+  run "$SECRETS_BIN" verify --bogus
+  [ "$status" -eq 1 ]
+  [[ "$output" == *"Unknown verify flag"* ]] || false
+}
+
+@test "verify: malformed manifest is refused" {
+  init_with_remote
+  create_project_dir verifymalformed
+  "$SECRETS_BIN" push >/dev/null 2>&1
+  printf 'not json{' > .secrets.json
+  run "$SECRETS_BIN" verify
+  [ "$status" -eq 1 ]
+  [[ "$output" == *"JSON"* ]] || false
+}
+
+@test "verify --all: empty store reports nothing to check (exit 0)" {
+  init_with_remote
+  cd "$WORK_DIR"
+  run "$SECRETS_BIN" verify --all
+  [ "$status" -eq 0 ]
+  [[ "$output" == *"empty"* ]] || false
+}