refactor: dedup external extractor + read guards, warn on legacy-pull nested blobs (EGB-701)

EGB-677 stage-1 structural cleanups, no behavior change for the v2 happy path:

1. cmd_which reuses _json_external_entries (the push/pull extractor) instead of
   its own duplicated jq @tsv projection, so `which` applies the same
   properties->gradle-properties normalization + skip-with-warning rules the
   sync path does and can't drift from it.
2. The two external-manifest read guards are factored into _json_readable
   (plain regular file, silent) / _legacy_readable (warn+skip a symlinked
   .secrets-files), shared by _external_entries_for_push/_pull.
3. The legacy (manifest-less) pull path now warns when nested <project>/<relpath>.age
   blobs exist that its non-recursive globs can't see (external/ excluded —
   pull_external_files handles those), so it never silently under-restores.

Tests: +4 in test/manifest.bats (normalized which display, malformed external
skipped by which, nested-blob warning fires, external-only no false warning).
Full suite green (286/286).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Brian Majewski 2026-06-18 10:08:13 -07:00
parent 57f1280262
commit 1ee096cc33
3 changed files with 132 additions and 23 deletions

View file

@ -76,8 +76,8 @@ secrets # CLI script (~2000 lines bash)
hooks/pre-commit # Pre-commit hook template
test/
secrets.bats # bats-core test suite (140 tests)
manifest.bats # EGB-677 .secrets.json manifest tests (78 tests)
migrate.bats # EGB-703 store-format-v2 migration tests (26 tests)
manifest.bats # EGB-677 .secrets.json manifest tests (83 tests)
migrate.bats # EGB-703 store-format-v2 migration tests (35 tests)
test_helper.bash # Shared setup/teardown
README.md # User-facing documentation
CLAUDE.md # This file
@ -107,7 +107,7 @@ The active store directory is picked by `resolve_store()` using these rules, hig
Key design decisions (all driven by /autoplan review):
- **Wire-in is at command scope** (`cmd_push`/`cmd_pull`), via `push_external_files` / `pull_external_files`, **not** inside `push_dir_to_project` / `pull_project_to_dir` (those loop per-workspace and `pull_project_to_dir` uses stdout as a data channel).
- **Storage:** blobs live in `$SECRETS_DIR/<project>/external/<slug>.gradle-properties.age`. The `external/` subdir keeps them out of the legacy non-recursive `*.age` / `.*.age` globs the dotenv `pull` path uses, so a dotenv pull can never decrypt an external blob into cwd. `cmd_rekey` and `cmd_list` instead walk the **entire** project tree (`find -type f`), so they cover both `external/<slug>.age` and nested manifest dotenv blobs (`<project>/<relpath>.age`) — rekey MUST recurse, or any nested/external blob is orphaned under the old key after rotation = data loss (EGB-677 regression test: "rekey re-encrypts a nested manifest dotenv blob"). `<slug>` = manifest path token with non-`[A-Za-z0-9._-]` chars → `_`, plus a `cksum` suffix of the original path so paths that clean to the same string (`a/b` vs `a_b`) don't collide. Machine-independent (derived from the committed manifest token, not the expanded path). `cmd_list --json` (EGB-699) emits the same recursive walk as a machine-readable object (`{store, projects[].entries[]}`, each entry `dotenv``path` or `external``subtype`+`path`) for tooling/CI (feeds EGB-671); jq assembles it so paths escape correctly and stdout stays pure JSON (the human store hint is suppressed; jq is a hard dep only in `--json` mode).
- **Storage:** blobs live in `$SECRETS_DIR/<project>/external/<slug>.gradle-properties.age`. The `external/` subdir keeps them out of the legacy non-recursive `*.age` / `.*.age` globs the dotenv `pull` path uses, so a dotenv pull can never decrypt an external blob into cwd. `cmd_rekey` and `cmd_list` instead walk the **entire** project tree (`find -type f`), so they cover both `external/<slug>.age` and nested manifest dotenv blobs (`<project>/<relpath>.age`) — rekey MUST recurse, or any nested/external blob is orphaned under the old key after rotation = data loss (EGB-677 regression test: "rekey re-encrypts a nested manifest dotenv blob"). **EGB-701 cleanups:** (1) the *legacy* (manifest-less) `pull` keeps its non-recursive globs but now **warns** when nested `<project>/<relpath>.age` blobs exist that those globs can't see (it excludes `external/`, which `pull_external_files` handles) — so a manifest-less pull never silently under-restores; the fix the warning points at is committing a `.secrets.json`. (2) `cmd_which`, push, and pull share one external extractor (`_json_external_entries`), so `which` applies the same `properties``gradle-properties` normalization and skip-with-warning rules the sync path does (it shows exactly what will sync, not a stale raw projection). (3) the two external-manifest read guards are factored into `_json_readable` (plain regular file, silent) / `_legacy_readable` (warn-and-skip on a symlinked legacy manifest). `<slug>` = manifest path token with non-`[A-Za-z0-9._-]` chars → `_`, plus a `cksum` suffix of the original path so paths that clean to the same string (`a/b` vs `a_b`) don't collide. Machine-independent (derived from the committed manifest token, not the expanded path). `cmd_list --json` (EGB-699) emits the same recursive walk as a machine-readable object (`{store, projects[].entries[]}`, each entry `dotenv``path` or `external``subtype`+`path`) for tooling/CI (feeds EGB-671); jq assembles it so paths escape correctly and stdout stays pure JSON (the human store hint is suppressed; jq is a hard dep only in `--json` mode).
- **Merge is pure bash, no `sed`/regex** (`merge_gradle_keys`): exact-string key comparison (avoids `beaconClerkPk` vs `beaconClerkPkTest` substring bug), value treated as opaque literal (survives `& \ /` in values). Updates a managed key in place at its first occurrence, collapses duplicates, appends new keys, preserves unrelated lines/comments/order. Continuation lines (trailing odd backslashes, tracked by `_trailing_bs_odd`) are never matched as keys. Atomic write: temp in the same dir → `chmod` to match (or `600` on create) → `mv`. Backs up to `<target>.secrets-bak` before each merge.
- **Properties separator parsing** (`_props_get`): key ends at the first `=`, `:`, or whitespace (after lstrip); handles `key=value`, `key = value`, `key:value`, `key value`; last definition wins.
- **Security:** the write target comes from a committed file, so `_validate_external_target_path` locks it down — basename must be `gradle.properties`, must resolve inside `$HOME` (deepest-existing-ancestor resolved, symlink target/parent refused, `..` rejected). This blocks a malicious manifest from appending decrypted keys to `~/.gitconfig`/`~/.bashrc`. `_parse_secrets_files_manifest` rejects shell metacharacters/control chars in path and keys (path allows `[A-Za-z0-9/._~-]` only; keys allow `[A-Za-z0-9._-]` + space), mirrors the `.secrets-store` posture (no shell expansion, symlinked manifest skipped).