feat(controls): enrich semantic retrieval query with finding intent + C5 live test #222

2026-07-22T07:06:33Z

sharang commented

2026-07-22 07:06:33 +00:00

Tuning driven by the C5 live run against the real 2,882-control master-controls corpus.

Problem C5 surfaced: the semantic pass retrieved on the code region alone. Two findings in the same auth.py (overlapping ±6 windows, both saturated with md5/password tokens) embedded alike and collapsed onto the same controls — a brute-force finding wrongly matched password-hashing controls.

Fix: build the retrieval query from the finding's title + description + region, so retrieval keys on what the finding is about, not just the ambient code. The raw region still goes to the grounded judge for snippet grounding (unchanged).

Verified live (same fixture, cached corpus index):

Finding	Before (region-only)	After (enriched)
Weak md5 hash	19874, 20683, 23149, 29985	mc-23149 rank #1 (eliminate weak unsalted hashes) + salted-hashing 21634
Login w/o brute-force protection	same 4, reordered	newly surfaces mc-19984 (brute_force_protection) + mc-23186 (account_lockout) — the correct controls, absent before

Also commits the gated live regression test (tests/c5_semantic_live.rs, #[ignore]d — not run by CI's --lib): an ingest-only check and the full semantic-stamping check against api-dev.

Green locally (clippy -D warnings, fmt, full suite).

🤖 Generated with Claude Code

Tuning driven by the **C5 live run** against the real 2,882-control master-controls corpus. **Problem C5 surfaced:** the semantic pass retrieved on the *code region alone*. Two findings in the same `auth.py` (overlapping ±6 windows, both saturated with md5/password tokens) embedded alike and collapsed onto the **same** controls — a *brute-force* finding wrongly matched password-*hashing* controls. **Fix:** build the retrieval query from the finding's **title + description + region**, so retrieval keys on what the finding is *about*, not just the ambient code. The raw region still goes to the grounded judge for snippet grounding (unchanged). **Verified live** (same fixture, cached corpus index): | Finding | Before (region-only) | After (enriched) | |---|---|---| | Weak md5 hash | 19874, 20683, 23149, 29985 | **mc-23149 rank #1** (eliminate weak unsalted hashes) + salted-hashing 21634 | | Login w/o brute-force protection | *same 4, reordered* | newly surfaces **mc-19984** (brute_force_protection) + **mc-23186** (account_lockout) — the correct controls, absent before | Also commits the **gated live regression test** (`tests/c5_semantic_live.rs`, `#[ignore]`d — not run by CI's `--lib`): an ingest-only check and the full semantic-stamping check against api-dev. Green locally (clippy -D warnings, fmt, full suite). 🤖 Generated with [Claude Code](https://claude.com/claude-code)

sharang added 1 commit 2026-07-22 07:06:34 +00:00

feat(controls): enrich semantic retrieval query with finding intent + C5 live test

CI / Check (push) Skipped

CI / Check (pull_request) Successful in 5m50s

Details

CI / Detect Changes (pull_request) Skipped

Details

CI / Deploy Agent (pull_request) Skipped

Details

CI / Deploy Dashboard (pull_request) Skipped

Details

CI / Deploy Docs (pull_request) Skipped

Details

CI / Deploy MCP (pull_request) Skipped

Details

31f1635ee8

Tuning from the C5 live run against the real 2,882-control master corpus. The
semantic pass retrieved on the code region alone, so two findings in one file
(overlapping windows, both md5/password tokens) collapsed onto the SAME controls —
a brute-force finding wrongly matched password-hashing controls.

Fix: build the retrieval query from the finding's title + description + region, so
retrieval keys on what the finding is *about*. The raw region still goes to the
judge for snippet grounding.

Verified live (same fixture, cached corpus index):
- 'Weak password hash (md5)'      -> mc-23149 (eliminate weak unsalted hashes) now RANK 1
- 'Login without brute-force prot' -> newly surfaces mc-19984 (brute_force_protection)
                                      + mc-23186 (account_lockout) — the correct controls,
                                      absent under region-only retrieval.

Also commits the gated live regression test (tests/c5_semantic_live.rs, #[ignore]d,
not run by CI's --lib): ingest-only + full semantic-stamping checks against api-dev.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

sharang merged commit a25d41c3e5 into main

2026-07-22 07:14:49 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: sharang/compliance-scanner-agent#222