feat: control-driven SAST — LUT + grounded LLM triage over tool findings #213

Merged

sharang merged 7 commits from feat/grounded-control-check into main 2026-07-21 09:01:52 +00:00

Conversation 0 Commits 7 Files Changed 17

+1453 -6

sharang commented 2026-07-21 08:38:02 +00:00

Owner

Copy Link

Copy Source

What

The control-driven SAST pipeline: deterministic tools detect, an authored LUT maps findings to compliance controls, and a grounded LLM triages (confirms + drops false positives). Pairs with the OSCAL loop (#210/#211/#212).

The pipeline

semgrep / gitleaks / syft-osv ─▶ raw finding                  (deterministic detection)
      control-map LUT          ─▶ controls_for(tool, cwe)      (authored, human-reviewed)
      grounded LLM judge       ─▶ confirm/refute, temp 0, must quote verbatim
      TriageOutcome            ─▶ Confirmed → stamp control_refs · FalsePositive → flag · Unmapped → leave

Wired as the orchestrator's control_triage stage, opt-in via BREAKPILOT_BASE_URL.

What's in it

• control-map crate — the authored control→scan LUT (40 CRA controls: 9 covered / 16 needs-tooling / 15 not-code-checkable) + query API, CI-wired.
• compliance-core::control_check — the grounding gate (verbatim-span; line recomputed from the match) + verdict cache. Finding.control_refs (serde default); the gate stamps it.
• controls/{judge,checker,triage,scan_triage} — LLM recognize stage (temp 0, fail-closed), the checker composition, the FP-filter, and the scan-pipeline integration (builds control specs from the ingested OSCAL catalog, reads a code window per finding, triages).

Design basis

Deterministic-detection + LLM-triage-only follows Snyk DeepCode / Copilot Autofix / ZeroFalse / IRIS. The verbatim-span grounding gate and the (control, span) cache are the parts that are novel for compliance findings (the finding is the audit artifact).

Testing

clippy -D warnings + cargo fmt --all --check + CI test cmd (-p core -p agent -p werkbank-exec -p control-map --lib, 297 + 51 + 4 + 27) all green. The pipeline is proven with a stub LLM: the grounding gate drops fabricated snippets, and triage drops mapped-but-refuted findings as false positives.

## What The **control-driven SAST pipeline**: deterministic tools detect, an authored LUT maps findings to compliance controls, and a grounded LLM triages (confirms + drops false positives). Pairs with the OSCAL loop (`#210`/`#211`/`#212`). ## The pipeline ``` semgrep / gitleaks / syft-osv ─▶ raw finding (deterministic detection) control-map LUT ─▶ controls_for(tool, cwe) (authored, human-reviewed) grounded LLM judge ─▶ confirm/refute, temp 0, must quote verbatim TriageOutcome ─▶ Confirmed → stamp control_refs · FalsePositive → flag · Unmapped → leave ``` Wired as the orchestrator's `control_triage` stage, **opt-in via `BREAKPILOT_BASE_URL`**. ## What's in it - **`control-map` crate** — the authored control→scan LUT (40 CRA controls: 9 covered / 16 needs-tooling / 15 not-code-checkable) + query API, CI-wired. - **`compliance-core::control_check`** — the grounding gate (verbatim-span; line recomputed from the match) + verdict cache. `Finding.control_refs` (serde default); the gate stamps it. - **`controls/{judge,checker,triage,scan_triage}`** — LLM recognize stage (temp 0, fail-closed), the checker composition, the FP-filter, and the scan-pipeline integration (builds control specs from the ingested OSCAL catalog, reads a code window per finding, triages). ## Design basis Deterministic-detection + LLM-triage-only follows Snyk DeepCode / Copilot Autofix / ZeroFalse / IRIS. The **verbatim-span grounding gate** and the **(control, span) cache** are the parts that are novel for *compliance* findings (the finding is the audit artifact). ## Testing `clippy -D warnings` + `cargo fmt --all --check` + CI test cmd (`-p core -p agent -p werkbank-exec -p control-map --lib`, **297 + 51 + 4 + 27**) all green. The pipeline is proven with a stub LLM: the grounding gate drops fabricated snippets, and triage drops mapped-but-refuted findings as false positives.

sharang added 6 commits 2026-07-21 08:38:03 +00:00

feat(core): grounded control-check backbone (ground gate + cache) ... bfeb3f041a

The deterministic spine of control-driven checking: turn a text control into a
finding via an LLM used as a grounded pattern-recognizer. ground() admits a
verdict only if its quoted snippet appears verbatim in the retrieved region and
recomputes the finding line from that match (the model's line is discarded), so
a fabricated snippet cannot survive. cache_key() makes verdicts reproducible.
Pure — no LLM, no IO. 4 lib tests incl. fabricated-snippet-dropped.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

feat(agent): control judge — LLM recognize stage (temp 0, verbatim, fail-closed) ... b0117078f3

ControlJudge trait + LlmControlJudge: judges one (control, region) with a closed
temperature-0 prompt that must quote the offending code VERBATIM; parsing fails
closed to non-violation (never a fabricated finding). Behind a trait so the
checker stays stub-testable. Its output is re-checked by the core grounding gate,
never trusted directly. 3 lib tests.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

feat(agent): grounded control checker — compose judge -> ground gate ... d1f42a1a83

GroundedControlChecker<J: ControlJudge>::check(spec, regions, repo_id) judges each
candidate region and keeps only findings that survive the core grounding gate.
Generic over the judge so tests drive it with a deterministic stub — the
recognize->ground path is proven without an LLM (grounded snippet kept, ungrounded
dropped, non-violation yields nothing). 2 lib tests.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

feat(control-map): deterministic control->scan LUT crate (CRA, 40 controls) ... b7534d1123

New standalone crate — the 'transcribing' layer that maps each control to the
static-scan step(s) that check it, or marks it needs_tooling / not_code_checkable.
Authored + human-reviewed: no LLM decides coverage. The LLM only triages the
tool's findings downstream (in the agent), never here.

- ControlMap / ControlEntry / ScanBinding / Coverage types + embedded JSON LUT
- query API: coverage(control), controls_for(tool, cwe), summary()
- CRA LUT: 40 controls -> 9 covered (semgrep/gitleaks/syft/osv) / 16 needs_tooling
  / 15 not_code_checkable
- wired into CI (clippy + test). 4 lib tests.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

feat(agent): control triage — LLM false-positive filter over tool findings ... f712ba1e60

ControlTriage composes the pipeline: a deterministic tool finding ->
controls_for(tool,cwe) [control-map LUT] -> grounded judge confirms/refutes ->
TriageOutcome { Unmapped | Confirmed([control ids]) | FalsePositive }. The LLM
enters ONLY here, as an FP filter over tool output (ZeroFalse/IRIS), never as the
detector; only judgments grounded to real code survive. Reuses the judge + core
ground gate + control-map. 3 lib tests (confirm/refute/unmapped).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

feat(agent): wire control triage into the scan pipeline (end-to-end SAST) ... 075a4cb81b

After the deterministic tools run, the orchestrator's new control_triage stage
stamps each finding with the compliance control(s) it is evidence for and flags
control false positives. triage_repo_findings builds control specs from the
ingested OSCAL catalog, reads a code window per finding, and runs ControlTriage
(control-map LUT -> grounded judge). Adds Finding.control_refs (serde default);
the ground gate stamps it. Opt-in via BREAKPILOT_BASE_URL. 2 region tests.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

sharang added 1 commit 2026-07-21 08:55:02 +00:00

fix(semgrep): extract CWE from list-form metadata (was always null) ... 753ecefaae

semgrep emits metadata.cwe as a LIST (e.g. "CWE-798: Use of Hard-coded
Credentials"), but the extractor called v.as_str() on it -> always None, so SAST
findings never carried a CWE at all (silently breaking control mapping AND the
CWE-based dedup). Handle list + bare-string forms and normalise to the CWE-NNN
id. Found by the live control-triage end-to-end test.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

sharang merged commit c6baf72c6d into main 2026-07-21 09:01:52 +00:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

Clear labels

P0-demo

CERTifAI P0 demo build tickets (CS-*)

bug

critical

enhancement

epic

Tracking issue spanning multiple sub-issues

high

infrastructure

onboarding

Multi-target onboarding wizard redesign

performance

security

v0.3.0

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

Benjamin_Boenisch sharang (Sharang Parnerkar)

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: sharang/compliance-scanner-agent#213