diff --git a/CHANGELOG.md b/CHANGELOG.md index 2325a2f..4a51e6a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,21 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to a four-digit MAJOR.MINOR.PATCH.MICRO version scheme. +## [0.7.2.0] - 2026-06-08 + +### Added + +- **`secrets list --json` (EGB-699)** — machine-readable listing for tooling and + CI. Emits a single JSON object on stdout: `{"store", "projects": [{"name", + "entries": [...]}]}`, where each entry self-describes via a `type` + discriminator — `{"type":"dotenv","path":}` or + `{"type":"external","subtype":"properties"|"file","path":}`. Reflects the + same recursive store walk as the human `list` (nested `/.age` + + `external/.age`). jq does the assembly so paths escape correctly; the + human store hint is suppressed so stdout stays pure JSON (notices → stderr). + jq is required only for `--json`. Feeds the EGB-671 install scripts, which need + to enumerate a cloned store programmatically instead of scraping the table. + ## [0.7.1.0] - 2026-06-08 ### Added diff --git a/CLAUDE.md b/CLAUDE.md index 7a0926c..608040a 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -75,7 +75,7 @@ Single bash script (`secrets`) with subcommands: init, push, pull, list, rm, rek secrets # CLI script (~2000 lines bash) hooks/pre-commit # Pre-commit hook template test/ - secrets.bats # bats-core test suite (133 tests) + secrets.bats # bats-core test suite (140 tests) manifest.bats # EGB-677 .secrets.json manifest tests (78 tests) migrate.bats # EGB-703 store-format-v2 migration tests (26 tests) test_helper.bash # Shared setup/teardown @@ -107,7 +107,7 @@ The active store directory is picked by `resolve_store()` using these rules, hig Key design decisions (all driven by /autoplan review): - **Wire-in is at command scope** (`cmd_push`/`cmd_pull`), via `push_external_files` / `pull_external_files`, **not** inside `push_dir_to_project` / `pull_project_to_dir` (those loop per-workspace and `pull_project_to_dir` uses stdout as a data channel). -- **Storage:** blobs live in `$SECRETS_DIR//external/.gradle-properties.age`. The `external/` subdir keeps them out of the legacy non-recursive `*.age` / `.*.age` globs the dotenv `pull` path uses, so a dotenv pull can never decrypt an external blob into cwd. `cmd_rekey` and `cmd_list` instead walk the **entire** project tree (`find -type f`), so they cover both `external/.age` and nested manifest dotenv blobs (`/.age`) — rekey MUST recurse, or any nested/external blob is orphaned under the old key after rotation = data loss (EGB-677 regression test: "rekey re-encrypts a nested manifest dotenv blob"). `` = manifest path token with non-`[A-Za-z0-9._-]` chars → `_`, plus a `cksum` suffix of the original path so paths that clean to the same string (`a/b` vs `a_b`) don't collide. Machine-independent (derived from the committed manifest token, not the expanded path). +- **Storage:** blobs live in `$SECRETS_DIR//external/.gradle-properties.age`. The `external/` subdir keeps them out of the legacy non-recursive `*.age` / `.*.age` globs the dotenv `pull` path uses, so a dotenv pull can never decrypt an external blob into cwd. `cmd_rekey` and `cmd_list` instead walk the **entire** project tree (`find -type f`), so they cover both `external/.age` and nested manifest dotenv blobs (`/.age`) — rekey MUST recurse, or any nested/external blob is orphaned under the old key after rotation = data loss (EGB-677 regression test: "rekey re-encrypts a nested manifest dotenv blob"). `` = manifest path token with non-`[A-Za-z0-9._-]` chars → `_`, plus a `cksum` suffix of the original path so paths that clean to the same string (`a/b` vs `a_b`) don't collide. Machine-independent (derived from the committed manifest token, not the expanded path). `cmd_list --json` (EGB-699) emits the same recursive walk as a machine-readable object (`{store, projects[].entries[]}`, each entry `dotenv`→`path` or `external`→`subtype`+`path`) for tooling/CI (feeds EGB-671); jq assembles it so paths escape correctly and stdout stays pure JSON (the human store hint is suppressed; jq is a hard dep only in `--json` mode). - **Merge is pure bash, no `sed`/regex** (`merge_gradle_keys`): exact-string key comparison (avoids `beaconClerkPk` vs `beaconClerkPkTest` substring bug), value treated as opaque literal (survives `& \ /` in values). Updates a managed key in place at its first occurrence, collapses duplicates, appends new keys, preserves unrelated lines/comments/order. Continuation lines (trailing odd backslashes, tracked by `_trailing_bs_odd`) are never matched as keys. Atomic write: temp in the same dir → `chmod` to match (or `600` on create) → `mv`. Backs up to `.secrets-bak` before each merge. - **Properties separator parsing** (`_props_get`): key ends at the first `=`, `:`, or whitespace (after lstrip); handles `key=value`, `key = value`, `key:value`, `key value`; last definition wins. - **Security:** the write target comes from a committed file, so `_validate_external_target_path` locks it down — basename must be `gradle.properties`, must resolve inside `$HOME` (deepest-existing-ancestor resolved, symlink target/parent refused, `..` rejected). This blocks a malicious manifest from appending decrypted keys to `~/.gitconfig`/`~/.bashrc`. `_parse_secrets_files_manifest` rejects shell metacharacters/control chars in path and keys (path allows `[A-Za-z0-9/._~-]` only; keys allow `[A-Za-z0-9._-]` + space), mirrors the `.secrets-store` posture (no shell expansion, symlinked manifest skipped). diff --git a/README.md b/README.md index 532ba11..e345d64 100644 --- a/README.md +++ b/README.md @@ -167,6 +167,7 @@ secrets clear | `secrets clear` | Delete plaintext secret files from the current directory | | `secrets run ` | Pull secrets, run a command, then clear secrets when it exits | | `secrets list` | Show all projects that have stored secrets | +| `secrets list --json` | Same listing as a machine-readable JSON object (`{store, projects[].entries[]}`, each entry `dotenv`/`external`) for tooling and CI. JSON goes to stdout; notices to stderr | | `secrets rm ` | Delete a project's secrets from the store | | `secrets rekey` | Generate a new encryption key and re-encrypt everything | | `secrets verify [project]` | Check the current project's `.secrets.json` against the store (missing/orphaned blobs) and decrypt every blob. `[project]` overrides the store directory name; the manifest is still read from the current directory | diff --git a/VERSION b/VERSION index 67085cc..9872478 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.7.1.0 +0.7.2.0 diff --git a/secrets b/secrets index b2f1559..3ad6c40 100755 --- a/secrets +++ b/secrets @@ -1744,9 +1744,21 @@ cmd_pull_workspaces() { } cmd_list() { + local json=0 + case "${1:-}" in + --json) json=1 ;; + "") ;; + *) die "Usage: secrets list [--json]" ;; + esac + resolve_store check_initialized + if [ "$json" -eq 1 ]; then + cmd_list_json + return + fi + local found=0 for dir in "$SECRETS_DIR"/*/; do [ -d "$dir" ] || continue @@ -1782,6 +1794,64 @@ cmd_list() { fi } +# EGB-699: machine-readable listing for tooling/CI (feeds EGB-671 install +# scripts). Contract: a single JSON object on stdout — +# {"store": "", "projects": [{"name", "entries": [...]}]} +# where each entry is {"type":"dotenv","path":} or +# {"type":"external","subtype":"properties"|"file","path":}. Mirrors the +# recursive store walk the human `list` uses (nested /.age + +# external/.age). jq does the assembly so paths are escaped correctly; +# stdout stays pure JSON (the human store hint is suppressed in this mode). +cmd_list_json() { + check_cmd jq + + { + for dir in "$SECRETS_DIR"/*/; do + [ -d "$dir" ] || continue + local project + project=$(basename "$dir") + [[ "$project" == .* ]] && continue + # Marker line so a project with zero blobs still appears (mirrors the + # human header), grouped via jq below. + printf 'project\t%s\n' "$project" + local f rel name + while IFS= read -r f; do + [ -f "$f" ] || continue + rel=${f#"$dir"} + rel=${rel%.age} + case "$rel" in + external/*) + name=${rel#external/} + case "$name" in + *.file) printf 'entry\t%s\texternal\tfile\t%s\n' "$project" "${name%.file}" ;; + *.properties) printf 'entry\t%s\texternal\tproperties\t%s\n' "$project" "${name%.properties}" ;; + *.gradle-properties) printf 'entry\t%s\texternal\tproperties\t%s\n' "$project" "${name%.gradle-properties}" ;; + *) printf 'entry\t%s\texternal\tunknown\t%s\n' "$project" "$name" ;; + esac + ;; + *) + printf 'entry\t%s\tdotenv\t\t%s\n' "$project" "$rel" + ;; + esac + done < <(find "$dir" -type f -name '*.age' | sort) + done + } | jq -R -n --arg store "$SECRETS_DIR" ' + [inputs | split("\t")] as $lines + | ($lines | map(select(.[0] == "project") | .[1]) | unique) as $names + | { + store: $store, + projects: ($names | map(. as $p | { + name: $p, + entries: [ $lines[] + | select(.[0] == "entry" and .[1] == $p) + | if .[2] == "external" + then { type: "external", subtype: .[3], path: .[4] } + else { type: "dotenv", path: .[4] } + end ] + })) + }' +} + cmd_rm() { check_cmd git resolve_store @@ -2464,6 +2534,7 @@ Usage: secrets clear -w|--workspaces Clear secrets from all workspaces in package.json secrets run [-w] Pull secrets, run command, clear secrets on exit secrets list List all projects and their secret files + secrets list --json Same listing as machine-readable JSON (for tooling/CI) secrets rm Remove a project's secrets from the repo secrets rekey Re-encrypt all secrets with a new key secrets verify [project] Check the manifest against the store + decrypt every blob @@ -2636,7 +2707,7 @@ case "${1:-help}" in cmd_run "$@" ;; add) cmd_add "${2:-}" ;; - list) cmd_list ;; + list) shift; cmd_list "$@" ;; rm) cmd_rm "${2:-}" ;; rekey) cmd_rekey ;; verify) shift; cmd_verify "$@" ;; diff --git a/test/secrets.bats b/test/secrets.bats index a75c0e6..3a72910 100644 --- a/test/secrets.bats +++ b/test/secrets.bats @@ -1763,3 +1763,84 @@ file_project() { [[ "$output" == *"Extracted 1 key"* ]] || false [[ "$output" == *"Encrypted file"* ]] || false } + +# ─── EGB-699: `list --json` machine-readable output ────────────────────── + +@test "EGB-699: list --json emits valid JSON with project and dotenv entry" { + init_with_remote + create_project_dir jproj + "$SECRETS_BIN" push jproj >/dev/null 2>&1 + run "$SECRETS_BIN" list --json + [ "$status" -eq 0 ] + # entire stdout parses as JSON + echo "$output" | jq -e . >/dev/null + # project is present + echo "$output" | jq -e '.projects[] | select(.name == "jproj")' >/dev/null + # .env shows up as a dotenv entry + echo "$output" | jq -e '.projects[] | select(.name == "jproj") + | .entries[] | select(.type == "dotenv" and .path == ".env")' >/dev/null +} + +@test "EGB-699: list --json includes a nested dotenv relpath" { + init_with_remote + create_project_dir nestjson + mkdir -p packages/web + echo "N=nested" > packages/web/.env.development + "$SECRETS_BIN" add packages/web/.env.development >/dev/null + "$SECRETS_BIN" push >/dev/null 2>&1 + run "$SECRETS_BIN" list --json + [ "$status" -eq 0 ] + echo "$output" | jq -e '.projects[] | select(.name == "nestjson") + | .entries[] | select(.type == "dotenv" and .path == "packages/web/.env.development")' >/dev/null +} + +@test "EGB-699: list --json marks an external properties entry with subtype" { + init_with_remote + gradle_src $'beaconClerkPkTest=pk_test_abc\n' + gradle_project gjson beaconClerkPkTest + "$SECRETS_BIN" push gjson >/dev/null 2>&1 + run "$SECRETS_BIN" list --json + [ "$status" -eq 0 ] + echo "$output" | jq -e '.projects[] | select(.name == "gjson") + | .entries[] | select(.type == "external" and .subtype == "properties")' >/dev/null +} + +@test "EGB-699: list --json marks an external file entry with subtype" { + init_with_remote + file_src + local dir="$WORK_DIR/fjson"; mkdir -p "$dir" + printf 'file ~/keystores/upload.keystore\n' > "$dir/.secrets-files" + cd "$dir" + "$SECRETS_BIN" push fjson >/dev/null 2>&1 + run "$SECRETS_BIN" list --json + [ "$status" -eq 0 ] + echo "$output" | jq -e '.projects[] | select(.name == "fjson") + | .entries[] | select(.type == "external" and .subtype == "file")' >/dev/null +} + +@test "EGB-699: list --json on an empty store emits an empty projects array" { + "$SECRETS_BIN" init >/dev/null 2>&1 + run "$SECRETS_BIN" list --json + [ "$status" -eq 0 ] + echo "$output" | jq -e '.projects == []' >/dev/null +} + +@test "EGB-699: list --json keeps stdout pure JSON (notices go to stderr)" { + # The non-default-store hint normally prints to stdout in human mode; under + # --json it must not, or it would corrupt the document. Capture stdout only. + init_with_remote + create_project_dir purejson + "$SECRETS_BIN" push purejson >/dev/null 2>&1 + local json + json=$("$SECRETS_BIN" list --json 2>/dev/null) + echo "$json" | jq -e . >/dev/null +} + +@test "EGB-699: list --json reports the active store path" { + init_with_remote + create_project_dir storejson + "$SECRETS_BIN" push storejson >/dev/null 2>&1 + run "$SECRETS_BIN" list --json + [ "$status" -eq 0 ] + echo "$output" | jq -e --arg s "$SECRETS_DIR" '.store == $s' >/dev/null +}