Compare commits

..
Author SHA1 Message Date
Sharang ParnerkarandClaude Fable 5 31f1635ee8 feat(controls): enrich semantic retrieval query with finding intent + C5 live test
CI / Check (push) Skipped
CI / Check (pull_request) Successful in 5m50s
CI / Detect Changes (pull_request) Skipped
CI / Deploy Agent (pull_request) Skipped
CI / Deploy Dashboard (pull_request) Skipped
CI / Deploy Docs (pull_request) Skipped
CI / Deploy MCP (pull_request) Skipped
Tuning from the C5 live run against the real 2,882-control master corpus. The
semantic pass retrieved on the code region alone, so two findings in one file
(overlapping windows, both md5/password tokens) collapsed onto the SAME controls —
a brute-force finding wrongly matched password-hashing controls.

Fix: build the retrieval query from the finding's title + description + region, so
retrieval keys on what the finding is *about*. The raw region still goes to the
judge for snippet grounding.

Verified live (same fixture, cached corpus index):
- 'Weak password hash (md5)'      -> mc-23149 (eliminate weak unsalted hashes) now RANK 1
- 'Login without brute-force prot' -> newly surfaces mc-19984 (brute_force_protection)
                                      + mc-23186 (account_lockout) — the correct controls,
                                      absent under region-only retrieval.

Also commits the gated live regression test (tests/c5_semantic_live.rs, #[ignore]d,
not run by CI's --lib): ingest-only + full semantic-stamping checks against api-dev.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-22 09:06:18 +02:00
12 changed files with 85 additions and 546 deletions
+8 -19
View File
@@ -7,13 +7,6 @@ on:
pull_request: pull_request:
env: env:
# registry + cosign creds via env, NOT inline ${{ }}: the Harbor robot
# username contains '$', which sh expands when interpolated into the
# script (robot$ci-push -> robot-push) => docker login unauthorized.
REGISTRY_USERNAME: ${{ secrets.REGISTRY_USERNAME }}
REGISTRY_PASSWORD: ${{ secrets.REGISTRY_PASSWORD }}
COSIGN_KEY: ${{ secrets.COSIGN_KEY }}
COSIGN_PASSWORD: ${{ secrets.COSIGN_PASSWORD }}
CARGO_TERM_COLOR: always CARGO_TERM_COLOR: always
RUSTFLAGS: "-D warnings" RUSTFLAGS: "-D warnings"
# Compile cache: sccache -> Hetzner S3 (breakpilot-sccache), runner-independent # Compile cache: sccache -> Hetzner S3 (breakpilot-sccache), runner-independent
@@ -214,12 +207,11 @@ jobs:
git init && git remote add origin "${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}.git" git init && git remote add origin "${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}.git"
git fetch --depth=1 origin "${GITHUB_SHA}" && git checkout FETCH_HEAD git fetch --depth=1 origin "${GITHUB_SHA}" && git checkout FETCH_HEAD
IMAGE=repo.meghsakha.com/certifai/compliance-agent IMAGE=repo.meghsakha.com/certifai/compliance-agent
echo "$REGISTRY_PASSWORD" | docker login repo.meghsakha.com -u "$REGISTRY_USERNAME" --password-stdin echo "${{ secrets.REGISTRY_PASSWORD }}" | docker login repo.meghsakha.com -u "${{ secrets.REGISTRY_USERNAME }}" --password-stdin
DOCKER_BUILDKIT=1 docker build --secret id=tramiton_token,env=TRAMITON_FETCH_TOKEN \ DOCKER_BUILDKIT=1 docker build --secret id=tramiton_token,env=TRAMITON_FETCH_TOKEN \
-f Dockerfile.agent -t "$IMAGE:latest" -t "$IMAGE:${GITHUB_SHA}" . -f Dockerfile.agent -t "$IMAGE:latest" -t "$IMAGE:${GITHUB_SHA}" .
docker push "$IMAGE:latest" && docker push "$IMAGE:${GITHUB_SHA}" docker push "$IMAGE:latest" && docker push "$IMAGE:${GITHUB_SHA}"
{ command -v cosign >/dev/null 2>&1 || curl -sSfLo /usr/local/bin/cosign https://github.com/sigstore/cosign/releases/download/v2.4.3/cosign-linux-amd64 || wget -qO /usr/local/bin/cosign https://github.com/sigstore/cosign/releases/download/v2.4.3/cosign-linux-amd64; } || echo "::warning::cosign fetch failed" command -v cosign >/dev/null 2>&1 || { curl -sSfLo /usr/local/bin/cosign https://github.com/sigstore/cosign/releases/download/v2.4.3/cosign-linux-amd64 && chmod +x /usr/local/bin/cosign; }
chmod +x /usr/local/bin/cosign 2>/dev/null || true
cosign sign --yes --key env://COSIGN_KEY "$IMAGE:latest" || echo "::warning::cosign failed" cosign sign --yes --key env://COSIGN_KEY "$IMAGE:latest" || echo "::warning::cosign failed"
PAYLOAD=$(printf '{"ref":"refs/heads/main","repository":{"full_name":"sharang/compliance-scanner-agent"},"head_commit":{"id":"%s","message":"deploy agent"}}' "${GITHUB_SHA}") PAYLOAD=$(printf '{"ref":"refs/heads/main","repository":{"full_name":"sharang/compliance-scanner-agent"},"head_commit":{"id":"%s","message":"deploy agent"}}' "${GITHUB_SHA}")
SIG=$(printf '%s' "$PAYLOAD" | openssl dgst -sha256 -hmac "${{ secrets.ORCA_WEBHOOK_SECRET }}" | awk '{print $2}') SIG=$(printf '%s' "$PAYLOAD" | openssl dgst -sha256 -hmac "${{ secrets.ORCA_WEBHOOK_SECRET }}" | awk '{print $2}')
@@ -241,12 +233,11 @@ jobs:
git init && git remote add origin "${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}.git" git init && git remote add origin "${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}.git"
git fetch --depth=1 origin "${GITHUB_SHA}" && git checkout FETCH_HEAD git fetch --depth=1 origin "${GITHUB_SHA}" && git checkout FETCH_HEAD
IMAGE=repo.meghsakha.com/certifai/compliance-dashboard IMAGE=repo.meghsakha.com/certifai/compliance-dashboard
echo "$REGISTRY_PASSWORD" | docker login repo.meghsakha.com -u "$REGISTRY_USERNAME" --password-stdin echo "${{ secrets.REGISTRY_PASSWORD }}" | docker login repo.meghsakha.com -u "${{ secrets.REGISTRY_USERNAME }}" --password-stdin
DOCKER_BUILDKIT=1 docker build --secret id=tramiton_token,env=TRAMITON_FETCH_TOKEN \ DOCKER_BUILDKIT=1 docker build --secret id=tramiton_token,env=TRAMITON_FETCH_TOKEN \
-f Dockerfile.dashboard -t "$IMAGE:latest" -t "$IMAGE:${GITHUB_SHA}" . -f Dockerfile.dashboard -t "$IMAGE:latest" -t "$IMAGE:${GITHUB_SHA}" .
docker push "$IMAGE:latest" && docker push "$IMAGE:${GITHUB_SHA}" docker push "$IMAGE:latest" && docker push "$IMAGE:${GITHUB_SHA}"
{ command -v cosign >/dev/null 2>&1 || curl -sSfLo /usr/local/bin/cosign https://github.com/sigstore/cosign/releases/download/v2.4.3/cosign-linux-amd64 || wget -qO /usr/local/bin/cosign https://github.com/sigstore/cosign/releases/download/v2.4.3/cosign-linux-amd64; } || echo "::warning::cosign fetch failed" command -v cosign >/dev/null 2>&1 || { curl -sSfLo /usr/local/bin/cosign https://github.com/sigstore/cosign/releases/download/v2.4.3/cosign-linux-amd64 && chmod +x /usr/local/bin/cosign; }
chmod +x /usr/local/bin/cosign 2>/dev/null || true
cosign sign --yes --key env://COSIGN_KEY "$IMAGE:latest" || echo "::warning::cosign failed" cosign sign --yes --key env://COSIGN_KEY "$IMAGE:latest" || echo "::warning::cosign failed"
PAYLOAD=$(printf '{"ref":"refs/heads/main","repository":{"full_name":"sharang/compliance-scanner-agent"},"head_commit":{"id":"%s","message":"deploy dashboard"}}' "${GITHUB_SHA}") PAYLOAD=$(printf '{"ref":"refs/heads/main","repository":{"full_name":"sharang/compliance-scanner-agent"},"head_commit":{"id":"%s","message":"deploy dashboard"}}' "${GITHUB_SHA}")
SIG=$(printf '%s' "$PAYLOAD" | openssl dgst -sha256 -hmac "${{ secrets.ORCA_WEBHOOK_SECRET }}" | awk '{print $2}') SIG=$(printf '%s' "$PAYLOAD" | openssl dgst -sha256 -hmac "${{ secrets.ORCA_WEBHOOK_SECRET }}" | awk '{print $2}')
@@ -266,11 +257,10 @@ jobs:
git init && git remote add origin "${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}.git" git init && git remote add origin "${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}.git"
git fetch --depth=1 origin "${GITHUB_SHA}" && git checkout FETCH_HEAD git fetch --depth=1 origin "${GITHUB_SHA}" && git checkout FETCH_HEAD
IMAGE=repo.meghsakha.com/certifai/compliance-docs IMAGE=repo.meghsakha.com/certifai/compliance-docs
echo "$REGISTRY_PASSWORD" | docker login repo.meghsakha.com -u "$REGISTRY_USERNAME" --password-stdin echo "${{ secrets.REGISTRY_PASSWORD }}" | docker login repo.meghsakha.com -u "${{ secrets.REGISTRY_USERNAME }}" --password-stdin
docker build -f Dockerfile.docs -t "$IMAGE:latest" -t "$IMAGE:${GITHUB_SHA}" . docker build -f Dockerfile.docs -t "$IMAGE:latest" -t "$IMAGE:${GITHUB_SHA}" .
docker push "$IMAGE:latest" && docker push "$IMAGE:${GITHUB_SHA}" docker push "$IMAGE:latest" && docker push "$IMAGE:${GITHUB_SHA}"
{ command -v cosign >/dev/null 2>&1 || curl -sSfLo /usr/local/bin/cosign https://github.com/sigstore/cosign/releases/download/v2.4.3/cosign-linux-amd64 || wget -qO /usr/local/bin/cosign https://github.com/sigstore/cosign/releases/download/v2.4.3/cosign-linux-amd64; } || echo "::warning::cosign fetch failed" command -v cosign >/dev/null 2>&1 || { curl -sSfLo /usr/local/bin/cosign https://github.com/sigstore/cosign/releases/download/v2.4.3/cosign-linux-amd64 && chmod +x /usr/local/bin/cosign; }
chmod +x /usr/local/bin/cosign 2>/dev/null || true
cosign sign --yes --key env://COSIGN_KEY "$IMAGE:latest" || echo "::warning::cosign failed" cosign sign --yes --key env://COSIGN_KEY "$IMAGE:latest" || echo "::warning::cosign failed"
PAYLOAD=$(printf '{"ref":"refs/heads/main","repository":{"full_name":"sharang/compliance-scanner-agent"},"head_commit":{"id":"%s","message":"deploy docs"}}' "${GITHUB_SHA}") PAYLOAD=$(printf '{"ref":"refs/heads/main","repository":{"full_name":"sharang/compliance-scanner-agent"},"head_commit":{"id":"%s","message":"deploy docs"}}' "${GITHUB_SHA}")
SIG=$(printf '%s' "$PAYLOAD" | openssl dgst -sha256 -hmac "${{ secrets.ORCA_WEBHOOK_SECRET }}" | awk '{print $2}') SIG=$(printf '%s' "$PAYLOAD" | openssl dgst -sha256 -hmac "${{ secrets.ORCA_WEBHOOK_SECRET }}" | awk '{print $2}')
@@ -292,12 +282,11 @@ jobs:
git init && git remote add origin "${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}.git" git init && git remote add origin "${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}.git"
git fetch --depth=1 origin "${GITHUB_SHA}" && git checkout FETCH_HEAD git fetch --depth=1 origin "${GITHUB_SHA}" && git checkout FETCH_HEAD
IMAGE=repo.meghsakha.com/certifai/compliance-mcp IMAGE=repo.meghsakha.com/certifai/compliance-mcp
echo "$REGISTRY_PASSWORD" | docker login repo.meghsakha.com -u "$REGISTRY_USERNAME" --password-stdin echo "${{ secrets.REGISTRY_PASSWORD }}" | docker login repo.meghsakha.com -u "${{ secrets.REGISTRY_USERNAME }}" --password-stdin
DOCKER_BUILDKIT=1 docker build --secret id=tramiton_token,env=TRAMITON_FETCH_TOKEN \ DOCKER_BUILDKIT=1 docker build --secret id=tramiton_token,env=TRAMITON_FETCH_TOKEN \
-f Dockerfile.mcp -t "$IMAGE:latest" -t "$IMAGE:${GITHUB_SHA}" . -f Dockerfile.mcp -t "$IMAGE:latest" -t "$IMAGE:${GITHUB_SHA}" .
docker push "$IMAGE:latest" && docker push "$IMAGE:${GITHUB_SHA}" docker push "$IMAGE:latest" && docker push "$IMAGE:${GITHUB_SHA}"
{ command -v cosign >/dev/null 2>&1 || curl -sSfLo /usr/local/bin/cosign https://github.com/sigstore/cosign/releases/download/v2.4.3/cosign-linux-amd64 || wget -qO /usr/local/bin/cosign https://github.com/sigstore/cosign/releases/download/v2.4.3/cosign-linux-amd64; } || echo "::warning::cosign fetch failed" command -v cosign >/dev/null 2>&1 || { curl -sSfLo /usr/local/bin/cosign https://github.com/sigstore/cosign/releases/download/v2.4.3/cosign-linux-amd64 && chmod +x /usr/local/bin/cosign; }
chmod +x /usr/local/bin/cosign 2>/dev/null || true
cosign sign --yes --key env://COSIGN_KEY "$IMAGE:latest" || echo "::warning::cosign failed" cosign sign --yes --key env://COSIGN_KEY "$IMAGE:latest" || echo "::warning::cosign failed"
PAYLOAD=$(printf '{"ref":"refs/heads/main","repository":{"full_name":"sharang/compliance-scanner-agent"},"head_commit":{"id":"%s","message":"deploy mcp"}}' "${GITHUB_SHA}") PAYLOAD=$(printf '{"ref":"refs/heads/main","repository":{"full_name":"sharang/compliance-scanner-agent"},"head_commit":{"id":"%s","message":"deploy mcp"}}' "${GITHUB_SHA}")
SIG=$(printf '%s' "$PAYLOAD" | openssl dgst -sha256 -hmac "${{ secrets.ORCA_WEBHOOK_SECRET }}" | awk '{print $2}') SIG=$(printf '%s' "$PAYLOAD" | openssl dgst -sha256 -hmac "${{ secrets.ORCA_WEBHOOK_SECRET }}" | awk '{print $2}')
+8 -7
View File
@@ -112,9 +112,9 @@ async fn build_specs(provider: &OscalControlsProvider) -> HashMap<String, Contro
/// the grounded judge decide whether the control holds there. Returns net-new /// the grounded judge decide whether the control holds there. Returns net-new
/// findings, each already tagged with its control and grounded to a real snippet. /// findings, each already tagged with its control and grounded to a real snippet.
/// ///
/// The orchestrator runs this when `breakpilot.grounded_control_checks` is set /// Gated: the orchestrator runs this only when `breakpilot.grounded_control_checks`
/// (on by default). Validated live; it covers the 8 absence-based CRA controls /// is set. Absence detection is the least deterministic path (the judge decides
/// (the judge decides presence/absence, grounded to a real snippet). /// presence/absence, not a syntactic pattern), so it stays off until tuned live.
pub async fn grounded_surface_findings( pub async fn grounded_surface_findings(
config: &AgentConfig, config: &AgentConfig,
llm: Arc<LlmClient>, llm: Arc<LlmClient>,
@@ -173,10 +173,11 @@ fn fetch_region(repo_path: &Path, file: &str, line: u32) -> Option<CandidateRegi
/// ~13.6k master-control corpus (which has no CWE to LUT on). Returns the number /// ~13.6k master-control corpus (which has no CWE to LUT on). Returns the number
/// of findings that gained a master-control ref. /// of findings that gained a master-control ref.
/// ///
/// The orchestrator runs this when `breakpilot.semantic_mapping` is set (on by /// Gated: the orchestrator runs this only when `breakpilot.semantic_mapping` is
/// default). The control embedding index is built once and cached to /// set (default off, flipped on once the master-controls catalog is live). The
/// `snapshot_dir` keyed by corpus hash ([`ControlIndex::load_or_build`]), so only /// control embedding index is built once and cached to `snapshot_dir` keyed by
/// the first scan after a catalog change pays the embedding cost. /// corpus hash ([`ControlIndex::load_or_build`]), so only the first scan after a
/// catalog change pays the embedding cost.
pub async fn semantic_stamp_findings( pub async fn semantic_stamp_findings(
config: &AgentConfig, config: &AgentConfig,
llm: Arc<LlmClient>, llm: Arc<LlmClient>,
+7 -40
View File
@@ -232,9 +232,9 @@ impl PipelineOrchestrator {
// Stage 5c: semantic control mapping — scale path for the master-controls // Stage 5c: semantic control mapping — scale path for the master-controls
// corpus (no CWE to LUT on): embed each finding's region, retrieve the // corpus (no CWE to LUT on): embed each finding's region, retrieve the
// nearest master controls, grounded-judge, and stamp confirmed refs. On by // nearest master controls, grounded-judge, and stamp confirmed refs. Gated
// default (validated live); the corpus embedding is cached so only the // (default off) as the corpus embedding + per-finding judging is the heavy
// first scan after a catalog change pays it. // path; enabled once verified live against a deployed master-controls catalog.
if self.config.breakpilot.semantic_mapping { if self.config.breakpilot.semantic_mapping {
self.update_phase(scan_run_id, "semantic_control_mapping") self.update_phase(scan_run_id, "semantic_control_mapping")
.await; .await;
@@ -256,8 +256,8 @@ impl PipelineOrchestrator {
// rate limiting, no security logging, no update-signature check) have no // rate limiting, no security logging, no update-signature check) have no
// syntactic pattern to match, so we retrieve the code surface each governs // syntactic pattern to match, so we retrieve the code surface each governs
// and let the grounded judge decide whether it holds, producing net-new // and let the grounded judge decide whether it holds, producing net-new
// findings already tagged + grounded. On by default (validated live); it // findings already tagged + grounded. Gated (default off): absence
// covers the 8 absence-based CRA controls. // detection is the least deterministic path, kept off until tuned live.
if self.config.breakpilot.grounded_control_checks { if self.config.breakpilot.grounded_control_checks {
self.update_phase(scan_run_id, "grounded_control_checks") self.update_phase(scan_run_id, "grounded_control_checks")
.await; .await;
@@ -277,10 +277,8 @@ impl PipelineOrchestrator {
} }
} }
// Dedup against existing findings: insert first-seen ones, and refresh the // Dedup against existing findings and insert new ones
// control mappings on ones we've seen before.
let mut new_count = 0u32; let mut new_count = 0u32;
let mut refreshed_count = 0u32;
let mut new_findings: Vec<Finding> = Vec::new(); let mut new_findings: Vec<Finding> = Vec::new();
for mut finding in all_findings { for mut finding in all_findings {
finding.scan_run_id = Some(scan_run_id.to_string()); finding.scan_run_id = Some(scan_run_id.to_string());
@@ -295,25 +293,8 @@ impl PipelineOrchestrator {
finding.id = result.inserted_id.as_object_id(); finding.id = result.inserted_id.as_object_id();
new_findings.push(finding); new_findings.push(finding);
new_count += 1; new_count += 1;
} else if !finding.control_refs.is_empty() {
// Re-scan refresh: a mapping pass (newly enabled or tuned) computed
// control_refs for a finding first seen before mapping ran. Persist
// them onto the existing row — the insert path alone never would.
self.db
.findings()
.update_one(
doc! { "fingerprint": &finding.fingerprint },
doc! { "$set": { "control_refs": finding.control_refs.clone() } },
)
.await?;
refreshed_count += 1;
} }
} }
if refreshed_count > 0 {
tracing::info!(
"[{repo_id}] Refreshed control_refs on {refreshed_count} existing findings"
);
}
// Remove stale SBOM entries for this repo before reinserting // Remove stale SBOM entries for this repo before reinserting
if !sbom_entries.is_empty() { if !sbom_entries.is_empty() {
@@ -586,21 +567,7 @@ impl PipelineOrchestrator {
let Some(path) = ingest_set.get(&a.id).and_then(|ia| ia.working_path.clone()) else { let Some(path) = ingest_set.get(&a.id).and_then(|ia| ia.working_path.clone()) else {
continue; continue;
}; };
let mut source_findings = crate::pipeline::plc::analyze_tree(&path, target_id); all_findings.extend(crate::pipeline::plc::analyze_tree(&path, target_id));
// Control mapping for the PLC path (run_plc_scan is separate from
// run_pipeline, which does its own mapping). PLC findings carry
// file_path/line/cwe, so the semantic pass reads each region under this
// source's `path` and stamps master-control refs. The LUT + grounded
// surface passes are code-pattern / CRA-specific and don't apply to
// IEC 61131-3 control logic, so only the semantic pass runs here.
crate::controls::semantic_stamp_findings(
&self.config,
self.llm.clone(),
&path,
&mut source_findings,
)
.await;
all_findings.extend(source_findings);
// Control-application SBOM: CODESYS libraries + runtime from a // Control-application SBOM: CODESYS libraries + runtime from a
// `.projectarchive` (uploaded, or committed in the working tree). // `.projectarchive` (uploaded, or committed in the working tree).
let archive = a let archive = a
-125
View File
@@ -1,125 +0,0 @@
//! C5 example 2 — exploratory (not a committed regression test). Four topically
//! distinct findings, to see whether tuned semantic retrieval maps each to the
//! right master-control family. Run:
//! export ... (LITELLM_* + BREAKPILOT_BASE_URL)
//! cargo test -p compliance-agent --test c5_example2 -- --ignored --nocapture
mod common;
use std::sync::Arc;
use compliance_agent::llm::LlmClient;
use compliance_core::config::BreakpilotConfig;
use compliance_core::models::finding::{Finding, Severity};
use compliance_core::models::scan::ScanType;
use secrecy::SecretString;
fn env(k: &str) -> String {
std::env::var(k).unwrap_or_else(|_| panic!("env {k} must be set"))
}
fn mk(file: &str, line: u32, title: &str, desc: &str) -> Finding {
let mut f = Finding::new(
"repo-c5b".into(),
format!("{file}:{line}"),
"semgrep".into(),
ScanType::Sast,
title.into(),
desc.into(),
Severity::High,
);
f.file_path = Some(file.into());
f.line_number = Some(line);
f
}
fn write(repo: &std::path::Path, rel: &str, body: &str) {
let p = repo.join(rel);
if let Some(parent) = p.parent() {
std::fs::create_dir_all(parent).unwrap();
}
std::fs::write(p, body).unwrap();
}
#[tokio::test]
#[ignore = "live: api-dev + LiteLLM"]
async fn c5b_varied_findings() {
let llm = Arc::new(LlmClient::new(
env("LITELLM_URL"),
SecretString::from(env("LITELLM_API_KEY")),
env("LITELLM_MODEL"),
env("LITELLM_EMBED_MODEL"),
));
let mut config = common::dev_config("mongodb://unused".into(), "c5b".into());
config.breakpilot = BreakpilotConfig {
base_url: Some(env("BREAKPILOT_BASE_URL")),
token: None,
snapshot_dir: std::env::temp_dir()
.join("c5-oscal-snap")
.to_string_lossy()
.into_owned(),
semantic_mapping: true,
grounded_control_checks: false,
};
let repo = std::env::temp_dir().join("c5b-fixture-repo");
let _ = std::fs::remove_dir_all(&repo);
write(
&repo,
"app/db.py",
"import sqlite3\n\ndef get_user(username):\n q = \"SELECT * FROM users WHERE name = '\" + username + \"'\"\n return conn.execute(q)\n",
);
write(
&repo,
"app/config.py",
"# service config\nAPI_KEY = \"sk_live_51H8xYz3kQ9v2bNmR7wT4uSpQ\"\nDB_HOST = \"db.internal\"\n",
);
write(
&repo,
"app/net.py",
"import requests\n\ndef fetch(url):\n return requests.get(url, verify=False, timeout=5)\n",
);
write(
&repo,
"app/ser.py",
"import pickle\n\ndef load_state(blob):\n return pickle.loads(blob)\n",
);
let mut findings = vec![
mk(
"app/db.py",
4,
"SQL injection via string-concatenated query",
"User input is concatenated directly into a SQL statement, allowing SQL injection.",
),
mk(
"app/config.py",
2,
"Hardcoded API credential in source",
"A live API key is hardcoded in source code instead of a secret store.",
),
mk(
"app/net.py",
4,
"TLS certificate verification disabled",
"requests is called with verify=False, disabling TLS certificate validation.",
),
mk(
"app/ser.py",
3,
"Insecure deserialization with pickle.loads",
"Untrusted data is deserialized with pickle.loads, allowing remote code execution.",
),
];
let tagged =
compliance_agent::controls::semantic_stamp_findings(&config, llm, &repo, &mut findings)
.await;
println!("\n=== C5 example 2: varied findings ===");
for f in &findings {
println!(" {:52} -> {:?}", f.title, f.control_refs);
}
println!("tagged: {tagged}/4");
let _ = std::fs::remove_dir_all(&repo);
assert!(tagged >= 1);
}
@@ -1,92 +0,0 @@
//! Live validation of the grounded surface path (Stage 5d) for absence-based CRA
//! controls. Ignored (hits api-dev CRA catalog + LiteLLM). Run:
//! export ... (LITELLM_* + BREAKPILOT_BASE_URL)
//! cargo test -p compliance-agent --test grounded_surface_live -- --ignored --nocapture
//!
//! Builds a fixture whose code surfaces trigger several absence-based controls
//! (no rate limiting, no security logging, unverified update) and checks that the
//! grounded checker produces control-tagged findings.
mod common;
use std::sync::Arc;
use compliance_agent::llm::LlmClient;
use compliance_core::config::BreakpilotConfig;
use secrecy::SecretString;
fn env(k: &str) -> String {
std::env::var(k).unwrap_or_else(|_| panic!("env {k} must be set"))
}
fn write(repo: &std::path::Path, rel: &str, body: &str) {
let p = repo.join(rel);
if let Some(parent) = p.parent() {
std::fs::create_dir_all(parent).unwrap();
}
std::fs::write(p, body).unwrap();
}
#[tokio::test]
#[ignore = "live: api-dev CRA catalog + LiteLLM"]
async fn grounded_surface_flags_absence_controls() {
let llm = Arc::new(LlmClient::new(
env("LITELLM_URL"),
SecretString::from(env("LITELLM_API_KEY")),
env("LITELLM_MODEL"),
env("LITELLM_EMBED_MODEL"),
));
let mut config = common::dev_config("mongodb://unused".into(), "grounded".into());
config.breakpilot = BreakpilotConfig {
base_url: Some(env("BREAKPILOT_BASE_URL")),
token: None,
snapshot_dir: std::env::temp_dir()
.join("grounded-snap")
.to_string_lossy()
.into_owned(),
semantic_mapping: false,
grounded_control_checks: true,
};
let repo = std::env::temp_dir().join("grounded-fixture-repo");
let _ = std::fs::remove_dir_all(&repo);
// cra-ai-11: login endpoint with no rate limiting / lockout
write(
&repo,
"app/auth.py",
"@app.route('/login', methods=['POST'])\ndef login():\n u = request.form['username']\n p = request.form['password']\n if authenticate(u, p):\n return redirect('/')\n return 'bad credentials', 401\n",
);
// cra-ai-24: privileged admin action with no security/audit logging
write(
&repo,
"app/admin.py",
"@app.route('/admin/delete_user', methods=['POST'])\ndef admin_delete_user():\n uid = request.form['uid']\n db.users.delete_one({'_id': uid})\n return 'ok', 200\n",
);
// cra-ai-28/29/30: firmware update applied without signature / checksum verification
write(
&repo,
"app/updater.py",
"def apply_firmware_update(url):\n blob = download(url)\n install_firmware(blob)\n reboot_device()\n",
);
let findings =
compliance_agent::controls::grounded_surface_findings(&config, llm, &repo, "repo-grounded")
.await;
println!("\n=== Grounded surface findings ({}) ===", findings.len());
for f in &findings {
println!(
" {:24} {}:{:?} {}",
f.control_refs.join(","),
f.file_path.as_deref().unwrap_or(""),
f.line_number,
f.title
);
}
let _ = std::fs::remove_dir_all(&repo);
assert!(
!findings.is_empty(),
"expected the grounded pass to flag at least one absence-based control"
);
}
+8 -9
View File
@@ -76,15 +76,14 @@ pub struct BreakpilotConfig {
/// Directory for catalog snapshots. /// Directory for catalog snapshots.
pub snapshot_dir: String, pub snapshot_dir: String,
/// Enable the master-controls **semantic** mapping pass (embed regions, /// Enable the master-controls **semantic** mapping pass (embed regions,
/// Enable the master-controls **semantic** mapping pass (embed regions, /// retrieve nearest controls, grounded-judge). Off by default: it is the
/// retrieve nearest controls, grounded-judge). On by default — validated live /// scale path and stays gated until verified live against a deployed
/// against the deployed master-controls catalog. Still a no-op unless /// master-controls catalog.
/// `base_url` is set and the catalog is reachable.
pub semantic_mapping: bool, pub semantic_mapping: bool,
/// Enable the **grounded surface** pass for absence-based controls (retrieve /// Enable the **grounded surface** pass for absence-based controls (retrieve
/// the code surface a control governs, judge whether it holds). On by default /// the code surface a control governs, judge whether it holds). Off by
/// — validated live; it covers the 8 absence-based CRA controls that no /// default: absence detection is the least deterministic path and stays gated
/// syntactic rule can. /// until tuned against live scans.
pub grounded_control_checks: bool, pub grounded_control_checks: bool,
} }
@@ -94,8 +93,8 @@ impl Default for BreakpilotConfig {
base_url: None, base_url: None,
token: None, token: None,
snapshot_dir: "/data/compliance-scanner/oscal".to_string(), snapshot_dir: "/data/compliance-scanner/oscal".to_string(),
semantic_mapping: true, semantic_mapping: false,
grounded_control_checks: true, grounded_control_checks: false,
} }
} }
} }
+9 -16
View File
@@ -42,19 +42,7 @@ async fn main() -> Result<(), Box<dyn std::error::Error>> {
let pool_for_factory = pool.clone(); let pool_for_factory = pool.clone();
let service = StreamableHttpService::new( let service = StreamableHttpService::new(
move || { move || Ok(ComplianceMcpServer::new(pool_for_factory.clone())),
// The factory runs in the request task, still inside the bearer
// middleware's `TENANT_ID` scope, and BEFORE rmcp spawns the
// session task (which would lose the task_local). So bind the
// tenant into the session's server instance here, once.
let tenant_id = auth::current_tenant_id().ok_or_else(|| {
std::io::Error::other("no tenant context when creating MCP session")
})?;
Ok(ComplianceMcpServer::new(
pool_for_factory.clone(),
tenant_id,
))
},
Arc::new(LocalSessionManager::default()), Arc::new(LocalSessionManager::default()),
StreamableHttpServerConfig::default(), StreamableHttpServerConfig::default(),
); );
@@ -81,11 +69,16 @@ async fn main() -> Result<(), Box<dyn std::error::Error>> {
tenant_id = %synth_tenant, tenant_id = %synth_tenant,
"stdio transport — using synthetic tenant id; DO NOT use in production" "stdio transport — using synthetic tenant id; DO NOT use in production"
); );
let server = ComplianceMcpServer::new(pool, synth_tenant); let server = ComplianceMcpServer::new(pool);
let transport = rmcp::transport::stdio(); let transport = rmcp::transport::stdio();
use rmcp::ServiceExt; use rmcp::ServiceExt;
let handle = server.serve(transport).await?; auth::TENANT_ID
handle.waiting().await?; .scope(synth_tenant, async {
let handle = server.serve(transport).await?;
handle.waiting().await?;
Ok::<_, Box<dyn std::error::Error>>(())
})
.await?;
} }
Ok(()) Ok(())
+13 -9
View File
@@ -2,33 +2,37 @@ use rmcp::{
handler::server::wrapper::Parameters, model::*, tool, tool_handler, tool_router, ServerHandler, handler::server::wrapper::Parameters, model::*, tool, tool_handler, tool_router, ServerHandler,
}; };
use crate::auth::current_tenant_id;
use crate::database::{Database, DatabasePool}; use crate::database::{Database, DatabasePool};
use crate::tools::{dast, findings, oscal, pentest, sbom}; use crate::tools::{dast, findings, oscal, pentest, sbom};
pub struct ComplianceMcpServer { pub struct ComplianceMcpServer {
pool: DatabasePool, pool: DatabasePool,
/// Tenant this session serves. Bound once at session creation (the HTTP
/// factory reads the bearer-set tenant while still in the request scope;
/// stdio passes a synthetic id) — NOT a per-request `task_local`, which is
/// lost across the `tokio::spawn` that runs the Streamable-HTTP session.
tenant_id: String,
#[allow(dead_code)] #[allow(dead_code)]
tool_router: rmcp::handler::server::router::tool::ToolRouter<Self>, tool_router: rmcp::handler::server::router::tool::ToolRouter<Self>,
} }
impl ComplianceMcpServer { impl ComplianceMcpServer {
/// The per-tenant `Database` for this session. /// Resolve the per-tenant `Database` from the bearer-set
/// `task_local`. Every tool handler calls this; missing context
/// surfaces as `internal_error` because it means the auth
/// middleware was misconfigured (handler ran without scope).
fn tenant_db(&self) -> Result<Database, rmcp::ErrorData> { fn tenant_db(&self) -> Result<Database, rmcp::ErrorData> {
Ok(self.pool.for_tenant_id(&self.tenant_id)) let tenant_id = current_tenant_id().ok_or_else(|| {
rmcp::ErrorData::internal_error(
"no tenant context — bearer middleware not in chain".to_string(),
None,
)
})?;
Ok(self.pool.for_tenant_id(&tenant_id))
} }
} }
#[tool_router] #[tool_router]
impl ComplianceMcpServer { impl ComplianceMcpServer {
pub fn new(pool: DatabasePool, tenant_id: String) -> Self { pub fn new(pool: DatabasePool) -> Self {
Self { Self {
pool, pool,
tenant_id,
tool_router: Self::tool_router(), tool_router: Self::tool_router(),
} }
} }
+24 -80
View File
@@ -52,16 +52,9 @@
{ {
"control": "cra-ai-6", "control": "cra-ai-6",
"title": "Integritaetspruefung", "title": "Integritaetspruefung",
"scans": [ "scans": [],
{ "note": "absence-based — no syntactic pattern; covered by the grounded surface check (retrieve surface + LLM judge), gated (BREAKPILOT_GROUNDED_CHECKS) pending live tuning",
"tool": "grounded-control-check", "status": "needs_tooling"
"scan_type": "code_review",
"cwe": [],
"rules": []
}
],
"note": "covered by the grounded surface check (retrieve code surface + grounded LLM judge decides presence/absence); validated live",
"status": "covered"
}, },
{ {
"control": "cra-ai-7", "control": "cra-ai-7",
@@ -145,30 +138,16 @@
{ {
"control": "cra-ai-11", "control": "cra-ai-11",
"title": "Brute-Force-Schutz", "title": "Brute-Force-Schutz",
"scans": [ "scans": [],
{ "note": "absence-based — no syntactic pattern; covered by the grounded surface check (retrieve surface + LLM judge), gated (BREAKPILOT_GROUNDED_CHECKS) pending live tuning",
"tool": "grounded-control-check", "status": "needs_tooling"
"scan_type": "code_review",
"cwe": [],
"rules": []
}
],
"note": "covered by the grounded surface check (retrieve code surface + grounded LLM judge decides presence/absence); validated live",
"status": "covered"
}, },
{ {
"control": "cra-ai-12", "control": "cra-ai-12",
"title": "Rollenbasierte Autorisierung", "title": "Rollenbasierte Autorisierung",
"scans": [ "scans": [],
{ "note": "absence-based — no syntactic pattern; covered by the grounded surface check (retrieve surface + LLM judge), gated (BREAKPILOT_GROUNDED_CHECKS) pending live tuning",
"tool": "grounded-control-check", "status": "needs_tooling"
"scan_type": "code_review",
"cwe": [],
"rules": []
}
],
"note": "covered by the grounded surface check (retrieve code surface + grounded LLM judge decides presence/absence); validated live",
"status": "covered"
}, },
{ {
"control": "cra-ai-13", "control": "cra-ai-13",
@@ -328,16 +307,9 @@
{ {
"control": "cra-ai-24", "control": "cra-ai-24",
"title": "Security-Logging", "title": "Security-Logging",
"scans": [ "scans": [],
{ "note": "absence-based — no syntactic pattern; covered by the grounded surface check (retrieve surface + LLM judge), gated (BREAKPILOT_GROUNDED_CHECKS) pending live tuning",
"tool": "grounded-control-check", "status": "needs_tooling"
"scan_type": "code_review",
"cwe": [],
"rules": []
}
],
"note": "covered by the grounded surface check (retrieve code surface + grounded LLM judge decides presence/absence); validated live",
"status": "covered"
}, },
{ {
"control": "cra-ai-25", "control": "cra-ai-25",
@@ -356,58 +328,30 @@
{ {
"control": "cra-ai-27", "control": "cra-ai-27",
"title": "Log-Integritaet und -Aufbewahrung", "title": "Log-Integritaet und -Aufbewahrung",
"scans": [ "scans": [],
{ "note": "absence-based — no syntactic pattern; covered by the grounded surface check (retrieve surface + LLM judge), gated (BREAKPILOT_GROUNDED_CHECKS) pending live tuning",
"tool": "grounded-control-check", "status": "needs_tooling"
"scan_type": "code_review",
"cwe": [],
"rules": []
}
],
"note": "covered by the grounded surface check (retrieve code surface + grounded LLM judge decides presence/absence); validated live",
"status": "covered"
}, },
{ {
"control": "cra-ai-28", "control": "cra-ai-28",
"title": "Sichere Update-Mechanismen", "title": "Sichere Update-Mechanismen",
"scans": [ "scans": [],
{ "note": "absence-based — no syntactic pattern; covered by the grounded surface check (retrieve surface + LLM judge), gated (BREAKPILOT_GROUNDED_CHECKS) pending live tuning",
"tool": "grounded-control-check", "status": "needs_tooling"
"scan_type": "code_review",
"cwe": [],
"rules": []
}
],
"note": "covered by the grounded surface check (retrieve code surface + grounded LLM judge decides presence/absence); validated live",
"status": "covered"
}, },
{ {
"control": "cra-ai-29", "control": "cra-ai-29",
"title": "Update-Authentizitaet", "title": "Update-Authentizitaet",
"scans": [ "scans": [],
{ "note": "absence-based — no syntactic pattern; covered by the grounded surface check (retrieve surface + LLM judge), gated (BREAKPILOT_GROUNDED_CHECKS) pending live tuning",
"tool": "grounded-control-check", "status": "needs_tooling"
"scan_type": "code_review",
"cwe": [],
"rules": []
}
],
"note": "covered by the grounded surface check (retrieve code surface + grounded LLM judge decides presence/absence); validated live",
"status": "covered"
}, },
{ {
"control": "cra-ai-30", "control": "cra-ai-30",
"title": "Update-Integritaet", "title": "Update-Integritaet",
"scans": [ "scans": [],
{ "note": "absence-based — no syntactic pattern; covered by the grounded surface check (retrieve surface + LLM judge), gated (BREAKPILOT_GROUNDED_CHECKS) pending live tuning",
"tool": "grounded-control-check", "status": "needs_tooling"
"scan_type": "code_review",
"cwe": [],
"rules": []
}
],
"note": "covered by the grounded surface check (retrieve code surface + grounded LLM judge decides presence/absence); validated live",
"status": "covered"
}, },
{ {
"control": "cra-ai-31", "control": "cra-ai-31",
+8 -12
View File
@@ -178,13 +178,11 @@ mod tests {
} }
#[test] #[test]
fn covered_and_not_checkable_are_populated() { fn every_bucket_is_represented() {
let s = ControlMap::cra().unwrap().summary(); let s = ControlMap::cra().unwrap().summary();
assert!(s.covered > 0); assert!(s.covered > 0);
assert!(s.needs_tooling > 0);
assert!(s.not_code_checkable > 0); assert!(s.not_code_checkable > 0);
// needs_tooling is now empty: every code-checkable control is either
// tool-covered or covered by the grounded surface pass.
assert_eq!(s.needs_tooling, 0);
} }
#[test] #[test]
@@ -217,16 +215,14 @@ mod tests {
} }
#[test] #[test]
fn coverage_after_grounded_promotion() { fn coverage_reflects_the_b_track_split() {
let s = ControlMap::cra().unwrap().summary(); let s = ControlMap::cra().unwrap().summary();
// 9 off-the-shelf + 4 custom-semgrep + 8 grounded surface controls (promoted // 9 already tool-covered + B1's 4 custom-semgrep controls.
// after the grounded path was validated live). assert_eq!(s.covered, 13);
assert_eq!(s.covered, 21); // The 8 grounded surface controls stay needs_tooling until live-tuned.
// Nothing left as needs_tooling — every code-checkable control is covered. assert_eq!(s.needs_tooling, 8);
assert_eq!(s.needs_tooling, 0); // B3 marked the 4 pure-architectural controls not code-checkable.
// The 4 pure-architectural controls remain not code-checkable.
assert_eq!(s.not_code_checkable, 19); assert_eq!(s.not_code_checkable, 19);
assert_eq!(s.total(), 40);
} }
#[test] #[test]
-1
View File
@@ -36,7 +36,6 @@ export default withMermaid(defineConfig({
{ text: 'Pentest Architecture', link: '/features/pentest-architecture' }, { text: 'Pentest Architecture', link: '/features/pentest-architecture' },
{ text: 'AI Chat', link: '/features/ai-chat' }, { text: 'AI Chat', link: '/features/ai-chat' },
{ text: 'Code Knowledge Graph', link: '/features/graph' }, { text: 'Code Knowledge Graph', link: '/features/graph' },
{ text: 'Compliance Control Mapping', link: '/features/control-mapping' },
{ text: 'MCP Integration', link: '/features/mcp-server' }, { text: 'MCP Integration', link: '/features/mcp-server' },
], ],
}, },
-136
View File
@@ -1,136 +0,0 @@
# Compliance Control Mapping
Control mapping connects the scanner's raw output — deterministic tool findings and the code itself — to the **compliance controls** each piece of evidence supports. A hardcoded credential stops being just "CWE-798 from semgrep" and becomes evidence for *"cra-ai-8: no default passwords"* and, at scale, master control *`mc-31761` hardcoded_secrets_detection*. Findings carry those references (`control_refs`) into the dashboard and out over the MCP server as OSCAL, so the compliance report is built from real, grounded findings rather than a questionnaire.
## The core principle: tools detect, the LLM judges
The design has one rule, borrowed from the ZeroFalse / IRIS line of research: **deterministic tools are the detectors; the LLM is only ever a grounded false-positive filter, never the thing that finds the issue.**
- A tool (semgrep, gitleaks, syft/osv, ZAP, nuclei) detects deterministically.
- An **authored, human-reviewed lookup table** (`control-map`) maps that detection to the control(s) it's evidence for.
- The LLM enters last, to *confirm or refute* the mapping against the actual code — and every surviving verdict is anchored to a verbatim snippet by the grounding gate.
This keeps hallucination out of detection. The LLM supplies cross-language, cross-stack pattern *recognition*; the surrounding machinery supplies determinism.
## Coverage model
Every control lands in one of three buckets, recorded in the `control-map` LUT (`control-map/data/cra_control_map.json`) and never decided by an LLM:
| Bucket | Meaning |
| --- | --- |
| `covered` | An existing tool's scan surfaces findings for this control |
| `needs_tooling` | Code-checkable, but no off-the-shelf tool digs it out — we author a detector or use the grounded surface check |
| `not_code_checkable` | A design/process property — out of static-scan scope |
For the **CRA** framework (40 controls) the split is **13 covered · 8 needs_tooling · 19 not_code_checkable**. The 16 originally-uncovered controls were resolved as a hybrid:
- **4 custom semgrep detectors** (`cra-ai-1`, `7`, `10`, `14`) — secure-by-default, weak password hashing, insecure session cookies, weak data-at-rest ciphers. Shipped in the binary and matched back to controls **by rule id** so a broad CWE can't over-attribute.
- **8 grounded surface checks** (`cra-ai-6`, `11`, `12`, `24`, `27`, `28`, `29`, `30`) — the absence-based controls (no rate limiting, no security logging, no update-signature check…) that have no syntactic pattern.
- **4 marked not_code_checkable** (`cra-ai-2`, `3`, `4`, `5`) — minimal attack surface, secure architecture, least privilege, tamper protection.
At scale, the **master-controls** corpus (breakpilot's deduped clusters, exported as OSCAL) currently provides **~2,882 code-checkable controls** (2,143 `network` + 739 `source_code`), matched semantically.
## The three mapping paths
```mermaid
flowchart TD
T[Deterministic tools\nsemgrep · gitleaks · syft/osv · ZAP] --> F[Findings]
F --> B["Stage 5b — LUT triage\ncontrols_for(tool, cwe / rule_id)"]
F --> C["Stage 5c — Semantic\nembed region+intent → top-K master controls"]
R[Repo source] --> D["Stage 5d — Grounded surface\nretrieve surface for absence-based controls"]
B --> J{{Grounded LLM judge\ntemp 0 · verbatim snippet}}
C --> J
D --> J
J -->|snippet grounds in region| S[Stamp control_refs]
J -->|refuted / ungrounded| X[Dropped]
```
All three paths converge on the same **grounded judge** and the same **grounding gate**. They differ only in how candidate (finding/region, control) pairs are produced.
### Stage 5b — deterministic LUT triage
The default path. A tool finding is matched to controls via `control_map.controls_for_finding(tool, cwe, rule_id)`; the judge then confirms each mapped control against the code region. Outcomes: `Confirmed([ids])` (stamp them), `FalsePositive` (drop the finding), or `Unmapped` (keep it untagged). Runs whenever `BREAKPILOT_BASE_URL` is set.
### Stage 5c — semantic retrieval (master-controls scale)
Master controls carry no CWE, so they can't be LUT-mapped. Instead we map by *similarity*: embed every control's requirement text once (cached), then for each finding retrieve the top-K nearest controls and hand them to the judge. Gated behind `BREAKPILOT_SEMANTIC_MAPPING` (default off). See [Semantic retrieval](#semantic-retrieval-in-detail).
### Stage 5d — grounded surface checks (absence-based controls)
Some controls are violated by an *absence* — no rate limiting on login, no security logging, no signature check on an update. There's no pattern for semgrep to match, so we deterministically retrieve the code **surface** the control governs (a login route, a logging setup, update/download code) by identifier/route terms, and let the judge decide whether the control holds there. Produces net-new, already-grounded findings. Gated behind `BREAKPILOT_GROUNDED_CHECKS` (default off).
## The grounding gate
No matter the path, a verdict becomes a finding only if it survives `compliance_core::control_check::ground`:
1. The judge runs at **temperature 0** with a closed prompt and must quote the offending code **verbatim** into `snippet`.
2. That snippet must appear **literally** in the retrieved region — otherwise the verdict is dropped.
3. The finding's line is **recomputed from the match**; the model's own line number is never trusted.
4. Verdicts are cached by content hash, so re-scans reproduce.
The model is allowed to be smart; it is never trusted.
## Semantic retrieval in detail
1. **Embed the corpus once.** Each control's requirement text is embedded with `bge-multilingual-gemma2` (3584-dim — multilingual matters, the master controls are in German while code is English). The embedding backend caps input arrays at 25 per request, so `embed()` chunks at 16; the whole `ControlIndex` is persisted to `snapshot_dir` keyed by a **corpus hash**, so only the first scan after a catalog change pays the embedding cost.
2. **Build the query from the finding's intent, not just the code.** The retrieval query is `finding.title + finding.description + region`, not the raw region. This is the single most important tuning: two findings in one file share overlapping windows and, on the code alone, embed alike and collapse onto the same controls. The finding's own words ("brute-force protection" vs "weak hash") carry the discriminating signal. The raw region still goes to the judge for grounding.
3. **Retrieve → judge → ground.** Top-K nearest by cosine, each judged against the region, each grounded.
## Worked examples
Both examples are from the live end-to-end verification (`c5_semantic_live.rs`) against the real ~2,882-control corpus.
### Example 1 — a small auth file (the tuning story)
Two findings in one `auth.py`: a weak `hashlib.md5(password)` hash and a login endpoint with no brute-force protection.
| Finding | Region-only retrieval | Intent-enriched retrieval |
| --- | --- | --- |
| Weak md5 hash | 19874, 20683, 23149, 29985 | **`mc-23149`** (eliminate weak unsalted hashes) at rank 1, + `mc-21634` salted hashing |
| Login w/o brute-force protection | *identical 4, reordered* | newly surfaces **`mc-19984`** brute_force_protection + **`mc-23186`** account_lockout |
Region-only retrieval gave both findings the *same* four password-hashing controls — the brute-force finding never found its real controls because its window is saturated with `password` tokens. Enriching the query with the finding's intent fixed it: the brute-force finding now pulls the correct rate-limiting / lockout controls out of the 2,882.
### Example 2 — four topically distinct vulnerabilities
| Finding | Top matched controls | Family |
| --- | --- | --- |
| SQL injection (string-concat query) | `sql_injection_prevention`, `sql_injection`, `parameterized_queries`, input_sanitization | input-validation ✓ |
| Hardcoded API credential | `hardcoded_secrets_detection`, credential_scanning, secrets_detection | credentials ✓ |
| TLS verification disabled (`verify=False`) | `https_enforcement`, `configuration_verification`, transport config | transport-encryption ✓ |
| Insecure deserialization (`pickle.loads`) | `deserialization`, `deserialization_testing`, `deserialization_security` | deserialization ✓ |
Every finding maps to its exact control family, with the most specific control often at the top, and the four sets are distinct.
## Known limitations
- **Absence findings are weak for semantic retrieval.** Similarity matches what code *is about*, not what it *lacks*; a "missing rate limiting" finding embeds like login code. This is exactly why the grounded surface path (Stage 5d) exists — it decides presence/absence at a retrieved surface rather than by embedding distance.
- **Generic catch-all controls co-occur.** `mc-20890 secure_development_security_code_review` appears in the top-K for many code-security findings because it is semantically near almost all of them. It's harmless (the judge grounds it, and it never crowds out the specific controls — the SQLi example didn't get it) but is a candidate for future down-weighting.
- **Corpus classification noise.** The master-controls `verification_method` classification is imperfect — e.g. a documentation control (`eu_declaration_accuracy`) is currently tagged `source_code`. That's a corpus-side data-quality issue, separate from the mapping engine.
## Emitting over MCP — closing the loop
Findings don't just land in the dashboard; they flow to breakpilot-compliance as OSCAL over the scanner's MCP server, so the compliance report is assembled from real, control-tagged findings.
- The MCP server exposes an **`oscal_assessment`** tool: given a `repo_id`, it emits a standard OSCAL 1.1 assessment-results document for that repo's findings — mapped findings target their controls via the stamped `control_refs`, and unmapped findings are reported **as-is** (as observations), so nothing is lost.
- breakpilot pulls it: `POST /v1/cra/oscal-from-scanner` calls `oscal_assessment` over MCP (Streamable HTTP + bearer) and consumes the pre-computed OSCAL — rather than pulling raw findings and re-assessing.
**Operational note — tenant context over HTTP.** The MCP server is multi-tenant; the bearer token resolves a tenant whose per-tenant database the tools query. rmcp's Streamable HTTP transport runs each session's tool calls in a `tokio::spawn`ed task, and `task_local`s do **not** cross a spawn — so binding the tenant in a per-request middleware `task_local` leaves tool handlers with no context (every call fails `no tenant context`). The fix is to bind the tenant to the **per-session server instance** at creation (the factory runs in the request scope before the spawn), not to a per-request task_local. Until this was fixed, the loop silently failed over HTTP and consumers fell back to demo data.
## Configuration
| Variable | Effect |
| --- | --- |
| `BREAKPILOT_BASE_URL` | breakpilot-compliance root; enables control ingest + all mapping passes. **Unset disables all control mapping** — findings are produced without `control_refs`. |
| `BREAKPILOT_SEMANTIC_MAPPING` | Stage 5c (semantic master-controls mapping). **Default on** (validated live). |
| `BREAKPILOT_GROUNDED_CHECKS` | Stage 5d (grounded surface checks). **Default on** (validated live). |
| `BREAKPILOT_SNAPSHOT_DIR` | Where OSCAL catalog snapshots and the cached control-embedding index live. |
The semantic and grounded passes default **on** now that both are validated live; each is still a no-op if `BREAKPILOT_BASE_URL` is unset or the catalog is unreachable, so they only ever add coverage. The live verifications live in `compliance-agent/tests/c5_semantic_live.rs` and `grounded_surface_live.rs` (ignored; run with `--ignored`).
## Appendix — the master-controls data pipeline
The master-controls corpus is produced by breakpilot-compliance and pulled as an OSCAL catalog from `GET /api/compliance/v1/oscal/catalog?framework=master-controls`. Two operational lessons are worth recording, because they cost real time to diagnose:
- **The catalog is served from `breakpilot_db`, not `postgres`.** Diagnostics run against the wrong database will look clean while the app serves something else entirely. Confirm the app's datname (`pg_stat_activity`) before trusting any count or `EXPLAIN`.
- **A constraint-less dump triplicated the master-control tables.** Restored without their PK/unique constraints, `master_controls` / `mc_verification` / `master_control_members` accumulated identical rows 3× (the same artifact migration `158` fixed for `doc_check_controls`). That inflated the catalog to ~26k dup'd controls and, with the indexes also missing, drove the export query to a >120s / 502. The fix (breakpilot migration `160`) ctid-dedups each table by its natural key and restores the constraints + indexes so it can't recur; the export query was also rewritten set-based (a single windowed pass instead of a per-row correlated subquery). After dedup: 41,850 → 13,950 master controls, catalog **25,938 → 2,882** code-checkable, endpoint **502 → 200 in ~3s**.