feat(agents): Semantic-Validator + Auto-Learning-Pattern-Library
CI / detect-changes (push) Successful in 5s
CI / branch-name (push) Has been skipped
CI / secret-scan (push) Has been skipped
CI / guardrail-integrity (push) Has been skipped
CI / dep-audit (push) Has been skipped
CI / sbom-scan (push) Has been skipped
CI / build-sha-integrity (push) Failing after 4s
CI / validate-canonical-controls (push) Successful in 11s
CI / loc-budget (push) Successful in 14s
CI / go-lint (push) Has been skipped
CI / nodejs-lint (push) Has been skipped
CI / python-lint (push) Has been skipped
CI / test-go (push) Has been skipped
CI / nodejs-build (push) Has been skipped
CI / test-python-dsms-gateway (push) Has been skipped
CI / iace-gt-coverage (push) Has been skipped
CI / test-python-backend (push) Successful in 29s
CI / test-python-document-crawler (push) Has been skipped
CI / detect-changes (push) Successful in 5s
CI / branch-name (push) Has been skipped
CI / secret-scan (push) Has been skipped
CI / guardrail-integrity (push) Has been skipped
CI / dep-audit (push) Has been skipped
CI / sbom-scan (push) Has been skipped
CI / build-sha-integrity (push) Failing after 4s
CI / validate-canonical-controls (push) Successful in 11s
CI / loc-budget (push) Successful in 14s
CI / go-lint (push) Has been skipped
CI / nodejs-lint (push) Has been skipped
CI / python-lint (push) Has been skipped
CI / test-go (push) Has been skipped
CI / nodejs-build (push) Has been skipped
CI / test-python-dsms-gateway (push) Has been skipped
CI / iace-gt-coverage (push) Has been skipped
CI / test-python-backend (push) Successful in 29s
CI / test-python-document-crawler (push) Has been skipped
Sprint 1.10 — Semantic-Validator (User-Vorgabe 2026-06-09):
- Statt unendlich Regex-Pattern fuer jede Schreibweise zu pflegen
(Tel/Telefon/Telefonnr/Phone/Fon/Funkanschluss/…), nutzen wir
bei MC-MISS einen LLM-Call: 'Ist die Pflichtangabe semantisch
doch da, nur unter abweichendem Label?'
- Bei LLM-Treffer: HIGH/MEDIUM-Finding wird zu LOW demoted,
Empfehlung wird zu 'Best-Practice Umbenennung: Management ->
Geschaeftsfuehrer' (mit STANDARD_LABELS-Mapping).
- 1 LLM-Call pro Slot statt N: cost-effizient.
Sprint 1.11 — Auto-Learning-Pattern-Library:
- Jedes Label das SVL findet wird in JSON persistiert:
/tmp/breakpilot/agent_learned_patterns.json
- Beim naechsten Run prueft der Agent zuerst gelernte Patterns
BEVOR er das HIGH-Finding emittiert -> kein LLM-Call mehr.
- Asymptotisch 0 LLM-Calls fuer haeufige Edge-Cases.
- Halluzinations-Schutz: prune_low_confidence() loescht Patterns
mit <0.5 Avg-Confidence nach 100 Beobachtungen.
- Idempotent: gleicher (field_id, label, agent) -> Counter +1.
Tests: 40/40 gruen (10 Pattern-Library + 7 SVL + 13 GT + 11 v2).
STANDARD_LABELS-Map deckt Impressum + Cookie-Policy. Spaeter
erweiterbar fuer DSE, AGB, Widerrufs-Agenten.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,119 @@
|
||||
"""Tests für den Semantic-Validator-Layer."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
|
||||
import pytest
|
||||
|
||||
from compliance.services.specialist_agents import AgentInput, ImpressumAgent
|
||||
from compliance.services.specialist_agents._semantic_validator import (
|
||||
STANDARD_LABELS,
|
||||
build_rename_action,
|
||||
standard_label,
|
||||
validate_present,
|
||||
)
|
||||
|
||||
|
||||
def _run(coro):
|
||||
return asyncio.get_event_loop().run_until_complete(coro)
|
||||
|
||||
|
||||
def test_standard_labels_cover_impressum_fields():
|
||||
"""Alle Impressum-Pflichtangaben müssen ein Standard-Label haben."""
|
||||
for fid in (
|
||||
"kontakt_telefon", "kontakt_email", "vertretungsberechtigte",
|
||||
"handelsregister", "ust_id", "name_anbieter",
|
||||
):
|
||||
assert fid in STANDARD_LABELS, f"missing standard label: {fid}"
|
||||
|
||||
|
||||
def test_build_rename_action_includes_old_and_new():
|
||||
a = build_rename_action("kontakt_telefon", "Telefonnr.")
|
||||
assert "Telefonnr." in a
|
||||
assert "Telefon" in a
|
||||
assert "Best-Practice" in a or "Umbenennung" in a
|
||||
|
||||
|
||||
def test_standard_label_falls_back_to_field_id():
|
||||
assert standard_label("kontakt_telefon") == "Telefon"
|
||||
assert standard_label("ghost_field") == "ghost_field"
|
||||
|
||||
|
||||
def test_validate_present_short_text_returns_empty():
|
||||
out = _run(validate_present(
|
||||
"x", [("kontakt_telefon", "Telefon")],
|
||||
))
|
||||
assert out == {}
|
||||
|
||||
|
||||
def test_validate_present_no_fields_returns_empty():
|
||||
out = _run(validate_present("Long impressum text" * 100, []))
|
||||
assert out == {}
|
||||
|
||||
|
||||
def test_semantic_demotion_high_to_low(monkeypatch):
|
||||
"""Wenn LLM bestätigt dass Pflichtangabe da ist: HIGH→LOW.
|
||||
|
||||
Test-Setup: Impressum-Text OHNE jegliche Telefon-Markierung
|
||||
(Pattern matched nicht). LLM-Mock behauptet aber 'Funkanschluss'
|
||||
wäre ein abweichendes Label für die Telefonnummer.
|
||||
"""
|
||||
from compliance.services.specialist_agents._escalation import (
|
||||
EscalationResult, SourceType,
|
||||
)
|
||||
from compliance.services.specialist_agents._base import EscalationLog
|
||||
|
||||
async def _fake_cascade(sys_prompt, user_prompt,
|
||||
expect_json=True, skip_ovh=False):
|
||||
# Nur auf den SVL-Prompt reagieren
|
||||
if "FEHLENDE PFLICHTANGABEN" not in user_prompt:
|
||||
return None, []
|
||||
log = EscalationLog(
|
||||
stage=SourceType.LLM_LOCAL, model="qwen2.5:7b",
|
||||
duration_ms=42, success=True,
|
||||
)
|
||||
res = EscalationResult(
|
||||
content='{"results":[]}',
|
||||
stage=SourceType.LLM_LOCAL,
|
||||
model="qwen2.5:7b",
|
||||
log=log,
|
||||
parsed={"results": [{
|
||||
"field_id": "kontakt_telefon",
|
||||
"found": True,
|
||||
"label_used": "Funkanschluss",
|
||||
"evidence": "Funkanschluss 0761/123456",
|
||||
"confidence": 0.9,
|
||||
}]},
|
||||
)
|
||||
return res, [log]
|
||||
monkeypatch.setattr(
|
||||
"compliance.services.specialist_agents._semantic_validator.cascade",
|
||||
_fake_cascade,
|
||||
)
|
||||
monkeypatch.setattr(
|
||||
"compliance.services.specialist_agents.impressum.agent.cascade",
|
||||
_fake_cascade,
|
||||
)
|
||||
# Text OHNE Telefon-Label → MC matched nicht → HIGH-Finding
|
||||
text = (
|
||||
"Beispiel GmbH\nMusterstr. 1\n12345 Berlin\n"
|
||||
"E-Mail: x@y.de\nFunkanschluss 0761/123456\n"
|
||||
"Geschäftsführer: Max Mustermann\n"
|
||||
"Handelsregister Berlin HRB 12345\n"
|
||||
"USt-IdNr: DE123456789"
|
||||
)
|
||||
agent = ImpressumAgent()
|
||||
out = _run(agent.evaluate(AgentInput(doc_type="impressum", text=text)))
|
||||
telefon_findings = [f for f in out.findings
|
||||
if f.field_id == "kontakt_telefon"]
|
||||
assert telefon_findings, "expected MC-miss → finding"
|
||||
f = telefon_findings[0]
|
||||
# Erwartet: SVL hat demoted zu LOW
|
||||
assert f.severity == "LOW", (
|
||||
f"Erwartet: LOW nach semantic-demote, got: {f.severity}. "
|
||||
f"Finding: {f}"
|
||||
)
|
||||
assert f.severity_reason == "label_mismatch"
|
||||
assert "Funkanschluss" in f.action
|
||||
assert "Telefon" in f.action
|
||||
Reference in New Issue
Block a user