fix(agents): Impressum+Cookie delegieren MC-Laden ans Main Tool — Scope-Filter + Maßnahmen
CI / detect-changes (push) Successful in 8s
CI / branch-name (push) Has been skipped
CI / guardrail-integrity (push) Has been skipped
CI / secret-scan (push) Has been skipped
CI / sbom-scan (push) Has been skipped
CI / build-sha-integrity (push) Failing after 4s
CI / iace-gt-coverage (push) Has been skipped
CI / test-python-backend (push) Successful in 30s
CI / test-python-document-crawler (push) Has been skipped
CI / test-python-dsms-gateway (push) Has been skipped
CI / dep-audit (push) Has been skipped
CI / validate-canonical-controls (push) Successful in 11s
CI / loc-budget (push) Successful in 14s
CI / go-lint (push) Has been skipped
CI / python-lint (push) Has been skipped
CI / nodejs-lint (push) Has been skipped
CI / nodejs-build (push) Has been skipped
CI / test-go (push) Has been skipped

Regression: Der v3-Agent-Pfad baute eine parallele MC-Pipeline
(_load_impressum_mcs / _load_cookie_mcs, Roh-SELECT) und lief damit an
allen Schutzmechanismen der Engine vorbei → GOV/Branchen-MCs als HIGH bei
OEM/Zulieferer, fremde MCs (Bestellbestätigung), und action=check_question
(Fragen statt Maßnahmen im Frontend).

- Agent delegiert MC-Laden an rag_document_checker._load_controls
  (P72-Scope, check_type='text', fits_doc_type/scope_requires).
- Subtraktives Sektor-Gate (SECTOR_PREFIXES) + Themen-Gate am Agent-Rand.
- action = konkrete Maßnahme (Imperativ) statt check_question.
- rag_document_checker: from __future__ import annotations (3.9-Import).
- mcs: Name-Pattern erkennt "Aktiengesellschaft" (OEM-Impressums).
- Tote GT-/Semantic-/Routes-Tests wiederbelebt (v3-Mismatch +
  agent.cascade-Patch-Target). Alle 72 Specialist-Tests grün.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Benjamin Admin
2026-06-09 11:30:16 +02:00
parent bd4882e143
commit 389e6de0c7
13 changed files with 641 additions and 135 deletions
@@ -14,6 +14,11 @@ Flow:
→ Returns structured results compatible with CheckItem format
"""
# Lazy annotations: dieses Modul nutzt PEP-604-Hints (z.B. `set[str] | None`)
# und muss auch auf Python 3.9 importierbar bleiben (lokale Tests / safe-
# import in compliance.api). Keine Pydantic-Modelle hier — daher unkritisch.
from __future__ import annotations
import logging
import os
import re
@@ -44,6 +44,19 @@ _SEV_TO_ENUM = {
}
def _build_measure(label: str, norm: str) -> str:
"""Maßnahme (Imperativ) statt Pruef-Frage als action. Das Tool
definiert Maßnahmen im Frontend — es stellt keine Fragen."""
base = (label or "").strip().rstrip(".")
if not base:
return ("Cookie-Angabe ergänzen und gegen die gesetzlichen "
"Vorgaben prüfen.")
msg = f"Cookie-Angabe ergänzen: {base}."
if norm:
msg += f" Rechtsgrundlage: {norm}."
return msg
class CookiePolicyAgent(BaseSpecialistAgent):
agent_id = "cookie_policy"
agent_version = "3.0"
@@ -77,6 +90,11 @@ class CookiePolicyAgent(BaseSpecialistAgent):
f"{telemetry.get('layer_0_field_hits', 0)} Pattern-Boosts · "
f"{telemetry.get('layer_0_boost_overrides', 0)} Boost-Overrides"
)
if telemetry.get("sector_dropped"):
notes_parts.append(
f"Scope-Filter: {telemetry['sector_dropped']} "
"Branchen-MCs entfernt"
)
seen: set[str] = set()
for r in results:
@@ -96,6 +114,9 @@ class CookiePolicyAgent(BaseSpecialistAgent):
if passed:
continue
label = r.get("label") or r.get("hint") or ""
norm_str = str(r.get("regulation") or "")
if r.get("article"):
norm_str = (norm_str + f" Art. {r.get('article')}").strip()
findings.append(Finding(
check_id=f"DBMC-{mc_id}",
agent=self.agent_id,
@@ -104,12 +125,9 @@ class CookiePolicyAgent(BaseSpecialistAgent):
severity=sev,
severity_reason="db_mc_failed",
title=str(label)[:200] or f"DB-MC {mc_id} nicht erfüllt",
norm=str(r.get("regulation") or "") +
(f" Art. {r.get('article')}"
if r.get("article") else ""),
norm=norm_str,
evidence="",
action=str(r.get("hint") or "")[:400]
or "Bitte gegen die Cookie-Pflichten prüfen.",
action=_build_measure(str(label), norm_str)[:400],
confidence=0.9,
sources=[EvidenceSource(
source_type=SourceType.MC,
@@ -1,7 +1,10 @@
"""Cookie-Policy v3-Pipeline — analog zu impressum/v3_engine.py.
Lädt 381 Cookie-MCs aus compliance.doc_check_controls (doc_type='cookie'),
ruft den deterministischen Keyword-Check + Embedding-Match + Boost-Override.
MC-Laden DELEGIERT an die Main-Tool-Engine (rag_document_checker._load_controls,
doc_type='cookie'): eine Quelle der Wahrheit inkl. P72-Scope, check_type='text'
und fits_doc_type/scope_requires. KEINE parallele Roh-Query mehr. Danach
deterministischer Keyword-Check + Embedding-Match + Boost-Override.
Zusaetzlich ein subtraktives Sektor-Gate (Branchen-Prefix) am Agent-Rand.
"""
from __future__ import annotations
@@ -13,9 +16,17 @@ from .regex_boost import boost_matches_db_mc, compute_regex_boosts
logger = logging.getLogger(__name__)
# Branchen-Prefix -> erwarteter Scope-Token (reuse aus dem Mail-V2-Filter).
try:
from compliance.services.mail_render_v2._scope_filter import (
SECTOR_PREFIXES,
)
except Exception: # pragma: no cover
SECTOR_PREFIXES = {}
async def run_v3_pipeline(
text: str, business_scope: set[str],
text: str, business_scope: set[str], db_url: str = "",
) -> tuple[list[dict[str, Any]], dict[str, Any]]:
if not text or len(text) < 100:
return [], {"reason": "text too short"}
@@ -24,8 +35,16 @@ async def run_v3_pipeline(
boosts = compute_regex_boosts(text)
boost_field_ids = sorted(boosts)
# Layer 1: alle 381 Cookie-MCs aus DB laden
controls = await _load_cookie_mcs()
# Layer 1: MC-Laden DELEGIERT an die Main-Tool-Engine (Scope-Schutz
# inklusive). Danach subtraktives Sektor-Gate am Agent-Rand.
try:
from compliance.services.rag_document_checker import _load_controls
controls = await _load_controls("cookie", db_url, 0, business_scope)
except Exception as e:
logger.warning("cookie v3 load via main-tool engine failed: %s", e)
controls = []
_normalize_criteria(controls)
controls, sector_dropped = _filter_sector(controls, business_scope)
results: list[dict[str, Any]] = []
if controls:
try:
@@ -91,51 +110,50 @@ async def run_v3_pipeline(
"layer_1_pass": layer_1_pass,
"layer_0_boost_overrides": boost_overrides,
"total_mcs": len(results),
"sector_dropped": sector_dropped,
}
return results, telemetry
async def _load_cookie_mcs() -> list[dict]:
"""Lädt alle 381 Cookie-MCs aus compliance.doc_check_controls."""
try:
import json
from classroom_engine.database import SessionLocal
from sqlalchemy import text as _sa_text
db = SessionLocal()
try:
rows = db.execute(_sa_text(
"SELECT id, control_id, control_uuid, title, regulation, "
" article, check_question, pass_criteria, "
" fail_criteria, severity "
"FROM compliance.doc_check_controls "
"WHERE doc_type='cookie' "
"ORDER BY severity DESC, title"
)).fetchall()
finally:
db.close()
out = []
for r in rows:
def _parse(v):
if isinstance(v, list): return v
if isinstance(v, str):
try:
j = json.loads(v)
return j if isinstance(j, list) else [v]
except Exception: return [v]
return []
out.append({
"id": str(r[0]),
"control_id": r[1],
"control_uuid": str(r[2]) if r[2] else "",
"title": r[3] or "",
"regulation": r[4] or "",
"article": r[5] or "",
"check_question": r[6] or "",
"pass_criteria": _parse(r[7]),
"fail_criteria": _parse(r[8]),
"severity": r[9] or "MEDIUM",
})
return out
except Exception as e:
logger.warning("_load_cookie_mcs failed: %s", e)
return []
def _normalize_criteria(controls: list[dict[str, Any]]) -> None:
"""asyncpg liefert JSONB-Spalten als Roh-String → zu Listen parsen."""
import json
for c in controls:
for key in ("pass_criteria", "fail_criteria"):
v = c.get(key)
if isinstance(v, list):
continue
if isinstance(v, str):
try:
parsed = json.loads(v)
c[key] = parsed if isinstance(parsed, list) else [v]
except Exception:
c[key] = [v] if v else []
else:
c[key] = []
def _filter_sector(
controls: list[dict[str, Any]],
business_scope: set[str],
) -> tuple[list[dict[str, Any]], int]:
"""Subtraktives Sektor-Gate: MCs deren control_id-Prefix eine Branche
bezeichnet (FIN/GOV/MED/INS/EDU/LEG/REL/POL), die NICHT im business_scope
liegt, werden verworfen — sonst tauchen z.B. GOV-MCs bei einem
OEM/Zulieferer als Finding auf. Reuse der SECTOR_PREFIXES."""
scope_lc = {s.lower() for s in (business_scope or set())}
kept: list[dict[str, Any]] = []
dropped = 0
for c in controls:
cid = c.get("control_id") or ""
prefix = cid.split("-")[0].upper() if "-" in cid else ""
required = SECTOR_PREFIXES.get(prefix)
if required and not (scope_lc & required):
dropped += 1
continue
kept.append(c)
if dropped:
logger.info("cookie v3 sector-filter: -%d Branchen-MCs", dropped)
return kept, dropped
@@ -54,6 +54,24 @@ _SEV_TO_ENUM = {
}
def _build_measure(label: str, norm: str) -> str:
"""Formuliert aus einer fehlenden Pflichtangabe eine konkrete Maßnahme
(Imperativ) statt die Pruef-Frage auszugeben.
Das Tool definiert Maßnahmen im Frontend — es stellt keine Fragen. Der
check_question-Text ('Ist X angegeben?') wird daher NICHT mehr als
action durchgereicht (sonst zeigt die Finding-Card eine Frage unter
'Pflicht-Maßnahme')."""
base = (label or "").strip().rstrip(".")
if not base:
return ("Pflichtangabe ergänzen und gegen die gesetzlichen "
"Vorgaben prüfen.")
msg = f"Pflichtangabe ergänzen: {base}."
if norm:
msg += f" Rechtsgrundlage: {norm}."
return msg
class ImpressumAgent(BaseSpecialistAgent):
agent_id = "impressum"
agent_version = "3.0"
@@ -96,6 +114,13 @@ class ImpressumAgent(BaseSpecialistAgent):
f"{telemetry.get('layer_0_field_hits', 0)} Pattern-Boosts · "
f"{telemetry.get('layer_0_boost_overrides', 0)} Boost-Overrides"
)
sec_drop = telemetry.get("sector_dropped", 0)
off_drop = telemetry.get("offtopic_dropped", 0)
if sec_drop or off_drop:
notes_parts.append(
f"Scope-Filter: {sec_drop} Branchen-MCs + "
f"{off_drop} themenfremde MCs entfernt"
)
# DB-MCs → Findings + Coverage
seen_db_mcs: set[str] = set()
@@ -116,6 +141,9 @@ class ImpressumAgent(BaseSpecialistAgent):
if passed:
continue
label = r.get("label") or r.get("hint") or ""
norm_str = str(r.get("regulation") or "")
if r.get("article"):
norm_str = (norm_str + f" Art. {r.get('article')}").strip()
findings.append(Finding(
check_id=f"DBMC-{mc_id}",
agent=self.agent_id,
@@ -124,12 +152,9 @@ class ImpressumAgent(BaseSpecialistAgent):
severity=sev,
severity_reason="db_mc_failed",
title=str(label)[:200] or f"DB-MC {mc_id} nicht erfüllt",
norm=str(r.get("regulation") or "") +
(f" Art. {r.get('article')}"
if r.get("article") else ""),
norm=norm_str,
evidence="",
action=str(r.get("hint") or "")[:400]
or "Bitte gegen die Pflichtangaben prüfen.",
action=_build_measure(str(label), norm_str)[:400],
confidence=0.9,
sources=[EvidenceSource(
source_type=SourceType.MC,
@@ -43,7 +43,8 @@ MCS: tuple[MC, ...] = (
# Label-free fallback: Firma (Rechtsform) + Adresse
re.compile(
r"\b[A-ZÄÖÜ][\w\-\& ]{1,80}?\s+"
r"(?:GmbH|AG|UG|KG|SE|GbR|OHG|Limited|Ltd|LLC)\s*"
r"(?:Aktiengesellschaft|GmbH|AG|UG|KG|SE|GbR|OHG|"
r"Limited|Ltd|LLC)\s*"
r"[\s\S]{0,400}?"
r"\b\d{5}\s+[A-ZÄÖÜ]",
re.IGNORECASE,
@@ -150,3 +150,44 @@ def boost_matches_db_mc(
if best is None or match_count > best[0]:
best = (match_count, field_id)
return best[1] if best else None
def criteria_on_topic(
pass_criteria: list | None,
fail_criteria: list | None = None,
min_hits: int = 2,
) -> bool:
"""Deterministischer Themen-Gate: gehoert eine DB-MC ueberhaupt ins
Impressum-Themenfeld?
Prueft ob die kombinierten pass/fail_criteria mindestens `min_hits`
UNTERSCHIEDLICHE Schluesselwoerter aus IRGENDEINEM der 12 Impressum-
Felder (BOOST_KEYWORDS) enthalten. Fremd-MCs (z.B. 'Bestellbestaetigung',
'behoerdliche Anzeige'), die faelschlich unter doc_type='impressum'
getaggt sind, haben keinen Themen-Ueberlapp und werden so aussortiert
— unabhaengig von DB-Sidecar/Klassifikation.
Konservativ in beide Richtungen:
- ein einzelner inzidenteller Treffer (z.B. 'E-Mail' in einer
Bestellbestaetigung) reicht NICHT -> >=2 verschiedene Stichwoerter.
- leere Kriterien -> on-topic behalten (lieber ein FP als eine echte
Pflichtangabe verlieren).
"""
crit_parts: list[str] = []
for c in (pass_criteria or []):
if c:
crit_parts.append(str(c).lower())
for c in (fail_criteria or []):
if c:
crit_parts.append(str(c).lower())
if not crit_parts:
return True
crit_text = " ".join(crit_parts)
hits: set[str] = set()
for kws in BOOST_KEYWORDS.values():
for kw in kws:
if kw in crit_text:
hits.add(kw)
if len(hits) >= min_hits:
return True
return False
@@ -1,17 +1,20 @@
"""Sprint-1.12 v3-Engine: läuft die volle 4-Layer-Pipeline auf einem
Impressum-Text:
"""v3-Engine: läuft die 4-Layer-Pipeline auf einem Impressum-Text.
Layer 0 — Regex-Boost (meine 12 Patterns aus mcs.py)
Layer 1 — Keyword-Match aus doc_check_controls.pass_criteria
(75 MCs in DB für Impressum)
Layer 2 — BGE-M3 Embedding-Match als Fallback (im
rag_document_checker integriert)
Layer 3 — Semantic-Validator (LLM) wenn auch Embedding nicht half
(im Agent angefasst, hier nur Ergebnisse durchgereicht)
Layer 0 — Regex-Boost (die 12 deterministischen Agent-Patterns)
Layer 1 — MC-Laden + Keyword-Match. Das LADEN delegiert an die
Main-Tool-Engine (rag_document_checker._load_controls):
eine Quelle der Wahrheit inkl. P72-Scope, check_type='text'
und fits_doc_type/scope_requires aus dem Sidecar. KEINE
parallele Roh-Query mehr.
Layer 2 — BGE-M3 Embedding-Match (mc_embedding_matcher, shared)
Layer 3 — Semantic-Validator (LLM) im Agent (hier nur durchgereicht)
Output: Liste Result-Dicts kompatibel mit rag_document_checker (passed,
severity, control_id, regulation, ...). Der Agent konvertiert sie dann
zu Finding-Objekten.
Zusätzlich am Agent-Rand: subtraktives Sektor-/Themen-Gate
(_filter_controls) — das Sektor-Gate (Branchen-Prefix) macht die Engine
nicht, es lebt sonst im Mail-V2-Report.
Output: Liste Result-Dicts kompatibel mit rag_document_checker. Der Agent
konvertiert sie zu Finding-Objekten.
"""
from __future__ import annotations
@@ -19,10 +22,25 @@ from __future__ import annotations
import logging
from typing import Any
from .regex_boost import boost_matches_db_mc, compute_regex_boosts
from .regex_boost import (
boost_matches_db_mc,
compute_regex_boosts,
criteria_on_topic,
)
logger = logging.getLogger(__name__)
# Branchen-Prefix -> erwarteter Scope-Token. Reuse aus dem Mail-V2-
# Scope-Filter, damit Agent-Pfad und Report-Pfad dieselbe Quelle nutzen
# (keine divergente Zweit-Logik). Import defensiv: faellt der Mail-Pfad
# weg, bleibt der Agent lauffaehig (ohne Sektor-Gate).
try:
from compliance.services.mail_render_v2._scope_filter import (
SECTOR_PREFIXES,
)
except Exception: # pragma: no cover - defensiver Fallback
SECTOR_PREFIXES = {}
async def run_v3_pipeline(
text: str,
@@ -43,11 +61,22 @@ async def run_v3_pipeline(
logger.info("v3 Layer-0 boosts: %d hits — %s",
len(boost_field_ids), boost_field_ids)
# Layer 1: lade ALLE 75 doc_check_controls für 'impressum' direkt
# aus DB. Sidecar-Klassifizierung wird bewusst übersprungen — der
# Agent soll auf der vollen MC-Liste arbeiten (Layer 3 LLM-Validator
# demoted Pattern-Misses zu LOW, sodass Breitenwirkung kein Risiko ist).
controls = await _load_impressum_mcs()
# Layer 1: MC-Laden DELEGIERT an die Main-Tool-Engine. Damit erbt der
# Agent automatisch deren Scope-Schutz (P72 canonical-scope,
# check_type='text', fits_doc_type/scope_requires) — genau die Filter,
# an denen die alte parallele Roh-Query vorbeilief.
try:
from compliance.services.rag_document_checker import _load_controls
controls = await _load_controls(
"impressum", db_url, 0, business_scope,
)
except Exception as e:
logger.warning("v3 load via main-tool engine failed: %s", e)
controls = []
_normalize_criteria(controls)
# Agent-Rand-Backstop (DB-unabhaengig): Sektor-Gate (Branchen-Prefix,
# macht die Engine nicht) + Themen-Gate (falls der Sidecar leer ist).
controls, drop_stats = _filter_controls(controls, business_scope)
results: list[dict[str, Any]] = []
if controls:
try:
@@ -122,55 +151,73 @@ async def run_v3_pipeline(
"layer_1_fail": layer_1_fail,
"layer_0_boost_overrides": boost_overrides,
"total_mcs": len(results),
"sector_dropped": drop_stats.get("sector_dropped", 0),
"offtopic_dropped": drop_stats.get("offtopic_dropped", 0),
}
logger.info("v3 telemetry: %s", telemetry)
return results, telemetry
async def _load_impressum_mcs() -> list[dict]:
"""Lädt alle Impressum-MCs aus compliance.doc_check_controls — ohne
Sidecar-Filter. v3_engine nimmt die volle Breite."""
try:
import json
from classroom_engine.database import SessionLocal
from sqlalchemy import text as _sa_text
db = SessionLocal()
try:
rows = db.execute(_sa_text(
"SELECT id, control_id, control_uuid, title, regulation, "
" article, check_question, pass_criteria, "
" fail_criteria, severity "
"FROM compliance.doc_check_controls "
"WHERE doc_type='impressum' "
"ORDER BY severity DESC, title"
)).fetchall()
finally:
db.close()
out: list[dict] = []
for r in rows:
def _parse(v):
if isinstance(v, list):
return v
if isinstance(v, str):
try:
j = json.loads(v)
return j if isinstance(j, list) else [v]
except Exception:
return [v]
return []
out.append({
"id": str(r[0]),
"control_id": r[1],
"control_uuid": str(r[2]) if r[2] else "",
"title": r[3] or "",
"regulation": r[4] or "",
"article": r[5] or "",
"check_question": r[6] or "",
"pass_criteria": _parse(r[7]),
"fail_criteria": _parse(r[8]),
"severity": r[9] or "MEDIUM",
})
return out
except Exception as e:
logger.warning("_load_impressum_mcs failed: %s", e)
return []
def _filter_controls(
controls: list[dict[str, Any]],
business_scope: set[str],
) -> tuple[list[dict[str, Any]], dict[str, int]]:
"""Subtraktiver Scope-Filter VOR der Bewertung.
1. Sektor-Gate — MCs deren control_id-Prefix eine Branche bezeichnet
(FIN/GOV/MED/INS/EDU/LEG/REL/POL), die NICHT im business_scope
liegt, werden verworfen. Fuer einen OEM/Zulieferer/Maschinenbauer
(kein Behoerden-/Finanz-/Medizin-Scope) fallen GOV/FIN/MED-MCs so
heraus — derselbe Mechanismus wie im Mail-V2-Report.
2. Themen-Gate — MCs ohne Impressum-Themenueberlapp werden verworfen
(faengt fremd-getaggte MCs wie 'Bestellbestaetigung').
Rein subtraktiv: entfernt nur falsch-positive Kandidaten, erzeugt nie
neue Findings.
"""
scope_lc = {s.lower() for s in (business_scope or set())}
kept: list[dict[str, Any]] = []
sector_dropped = 0
offtopic_dropped = 0
for c in controls:
cid = c.get("control_id") or ""
prefix = cid.split("-")[0].upper() if "-" in cid else ""
required = SECTOR_PREFIXES.get(prefix)
if required and not (scope_lc & required):
sector_dropped += 1
continue
if not criteria_on_topic(c.get("pass_criteria"),
c.get("fail_criteria")):
offtopic_dropped += 1
continue
kept.append(c)
if sector_dropped or offtopic_dropped:
logger.info(
"v3 scope-filter: -%d Branchen-MCs, -%d themenfremde MCs "
"(scope=%s)", sector_dropped, offtopic_dropped,
sorted(scope_lc) or "leer",
)
return kept, {
"sector_dropped": sector_dropped,
"offtopic_dropped": offtopic_dropped,
}
def _normalize_criteria(controls: list[dict[str, Any]]) -> None:
"""asyncpg liefert JSONB-Spalten (pass_criteria/fail_criteria) als
Roh-String. In echte Listen parsen, damit Sektor-/Themen-Gate und
der Boost-Layer Element-weise (nicht Zeichen-weise) iterieren."""
import json
for c in controls:
for key in ("pass_criteria", "fail_criteria"):
v = c.get(key)
if isinstance(v, list):
continue
if isinstance(v, str):
try:
parsed = json.loads(v)
c[key] = parsed if isinstance(parsed, list) else [v]
except Exception:
c[key] = [v] if v else []
else:
c[key] = []