feat: add Golden Test Suite v1 (40 regression tests for Pass 0b pipeline)
8 categories: duplicate explosion, compound split, negative obligations, container detection, framework decomposition, evidence leakage, scope dimension, title quality. Includes global quality gates. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Benjamin Admin
353
control-pipeline/tests/golden_controls.yaml
Normal file
353
control-pipeline/tests/golden_controls.yaml
Normal file
@@ -0,0 +1,353 @@
|
||||
# Golden Test Suite v1 — 40 Regression Tests for Pass 0b Pipeline
|
||||
# Tests atomic control generation quality, deduplication, and classification.
|
||||
version: "1.0"
|
||||
purpose: "Regression tests for obligation-to-atomic-control pipeline"
|
||||
|
||||
global_quality_gates:
|
||||
max_controls_per_single_obligation: 6
|
||||
max_duplicate_title_count: 1
|
||||
truncated_titles_allowed: false
|
||||
evidence_as_atomic_control_allowed: false
|
||||
framework_container_as_atomic_allowed: false
|
||||
composite_control_as_atomic_allowed: false
|
||||
|
||||
tests:
|
||||
|
||||
# A. Duplicate Explosion (5)
|
||||
|
||||
- id: GT-DEDUPE-001
|
||||
name: API Rate Limiting duplicates merge
|
||||
category: duplicate_explosion
|
||||
input: "Rate-Limiting für alle API-Endpunkte einführen."
|
||||
expected:
|
||||
create: ["Rate-Limiting für API-Endpunkte technisch durchgesetzt"]
|
||||
max_duplicate_title_count: 1
|
||||
|
||||
- id: GT-DEDUPE-002
|
||||
name: Payload limit duplicate variants merge
|
||||
category: duplicate_explosion
|
||||
input: "Maximale Payload-Größen definieren und technisch durchsetzen."
|
||||
expected:
|
||||
create: ["Maximale Payload-Größen definiert", "Payload-Grenzen technisch durchgesetzt"]
|
||||
max_duplicate_title_count: 1
|
||||
|
||||
- id: GT-DEDUPE-003
|
||||
name: Session cookie duplicate variants merge
|
||||
category: duplicate_explosion
|
||||
input: "Session-Cookies müssen Secure, HttpOnly und SameSite setzen."
|
||||
expected:
|
||||
create: ["Sichere Cookie-Attribute für Session-Cookies gesetzt"]
|
||||
max_duplicate_title_count: 1
|
||||
|
||||
- id: GT-DEDUPE-004
|
||||
name: MFA repeated wording merges
|
||||
category: duplicate_explosion
|
||||
input: "Privilegierte Accounts müssen MFA nutzen. MFA ist für Administrationskonten verpflichtend."
|
||||
expected:
|
||||
create: ["MFA für privilegierte Accounts aktiviert"]
|
||||
max_duplicate_title_count: 1
|
||||
|
||||
- id: GT-DEDUPE-005
|
||||
name: SBOM variants merge
|
||||
category: duplicate_explosion
|
||||
input: "Ein SBOM ist zu führen. Ein Komponentenverzeichnis ist zu pflegen."
|
||||
expected:
|
||||
create: ["Verzeichnis aller Drittkomponenten geführt"]
|
||||
max_duplicate_title_count: 1
|
||||
|
||||
# B. Compound Action Split (5)
|
||||
|
||||
- id: GT-SPLIT-001
|
||||
name: Define and enforce API payload limits
|
||||
category: compound_action_split
|
||||
input: "Maximale Payload-Größen für API-Anfragen und -Antworten definieren und technisch durchsetzen."
|
||||
expected:
|
||||
create:
|
||||
- {action_type: define, object: api_payload_limits}
|
||||
- {action_type: implement, object: api_payload_enforcement}
|
||||
|
||||
- id: GT-SPLIT-002
|
||||
name: Create and implement maintenance policy
|
||||
category: compound_action_split
|
||||
input: "Eine Wartungsrichtlinie ist zu erstellen und zu implementieren."
|
||||
expected:
|
||||
create: ["Wartungsrichtlinie definiert", "Wartungsrichtlinie umgesetzt"]
|
||||
|
||||
- id: GT-SPLIT-003
|
||||
name: Identify assess remediate monitor vulnerabilities
|
||||
category: compound_action_split
|
||||
input: "Schwachstellen in Drittkomponenten sind zu identifizieren, zu bewerten, zu adressieren und zu überwachen."
|
||||
expected:
|
||||
create:
|
||||
- "Schwachstellen in Drittkomponenten identifiziert"
|
||||
- "Schwachstellen in Drittkomponenten bewertet"
|
||||
- "Schwachstellen in Drittkomponenten behandelt"
|
||||
- "Schwachstellen in Drittkomponenten überwacht"
|
||||
|
||||
- id: GT-SPLIT-004
|
||||
name: Report and document incident
|
||||
category: compound_action_split
|
||||
input: "Sicherheitsvorfälle sind zu melden und zu dokumentieren."
|
||||
expected:
|
||||
create: ["Sicherheitsvorfälle gemeldet"]
|
||||
evidence_attach: ["Incident-Dokumentation"]
|
||||
|
||||
- id: GT-SPLIT-005
|
||||
name: Test and validate control
|
||||
category: compound_action_split
|
||||
input: "Rate-Limits und Timeouts sind zu testen und zu validieren."
|
||||
expected:
|
||||
create: ["Rate-Limits und Timeouts getestet", "Rate-Limits und Timeouts validiert"]
|
||||
|
||||
# C. Negative Obligations (5)
|
||||
|
||||
- id: GT-NEG-001
|
||||
name: Sensitive data not in tokens or URLs
|
||||
category: negative_obligation_handling
|
||||
input: "Session-Management-Mechanismen dürfen keine sensiblen Daten im Token oder in der URL übertragen."
|
||||
expected:
|
||||
create: ["Sensible Daten nicht in Session-Token oder URLs übertragen"]
|
||||
forbidden: ["sensible Daten im Token umgesetzt"]
|
||||
|
||||
- id: GT-NEG-002
|
||||
name: SMS OTP excluded
|
||||
category: negative_obligation_handling
|
||||
input: "SMS-basierte OTP sind für privilegierte Accounts nicht zulässig."
|
||||
expected:
|
||||
create: ["SMS-OTP für privilegierte Accounts ausgeschlossen"]
|
||||
action_type: exclude
|
||||
|
||||
- id: GT-NEG-003
|
||||
name: Assertion reuse prevented
|
||||
category: negative_obligation_handling
|
||||
input: "Assertions bei föderierter Re-Authentifizierung dürfen nicht wiederverwendet werden."
|
||||
expected:
|
||||
create: ["Wiederverwendung föderierter Assertions verhindert"]
|
||||
action_type: prevent
|
||||
|
||||
- id: GT-NEG-004
|
||||
name: No weak crypto
|
||||
category: negative_obligation_handling
|
||||
input: "Veraltete kryptografische Algorithmen dürfen nicht verwendet werden."
|
||||
expected:
|
||||
create: ["Veraltete kryptografische Algorithmen ausgeschlossen"]
|
||||
action_type: exclude
|
||||
|
||||
- id: GT-NEG-005
|
||||
name: No secrets in logs
|
||||
category: negative_obligation_handling
|
||||
input: "Geheimnisse dürfen nicht in Logs gespeichert werden."
|
||||
expected:
|
||||
create: ["Geheimnisse in Logs verhindert"]
|
||||
action_type: prevent
|
||||
|
||||
# D. Container Detection (5)
|
||||
|
||||
- id: GT-CONTAINER-001
|
||||
name: Secure session management is composite
|
||||
category: container_control_detection
|
||||
input: "Sichere Sitzungsverwaltung muss umgesetzt werden."
|
||||
expected:
|
||||
routing_type: composite
|
||||
atomic_control_created: false
|
||||
|
||||
- id: GT-CONTAINER-002
|
||||
name: Token protection is composite
|
||||
category: container_control_detection
|
||||
input: "Token-Schutz muss umgesetzt werden."
|
||||
expected:
|
||||
routing_type: composite
|
||||
atomic_control_created: false
|
||||
|
||||
- id: GT-CONTAINER-003
|
||||
name: Third-party due diligence is composite
|
||||
category: container_control_detection
|
||||
input: "Sorgfaltspflichten für Drittkomponenten müssen umgesetzt werden."
|
||||
expected:
|
||||
routing_type: composite
|
||||
atomic_control_created: false
|
||||
|
||||
- id: GT-CONTAINER-004
|
||||
name: AI risk management system is composite
|
||||
category: container_control_detection
|
||||
input: "Ein Risikomanagementsystem für KI-Systeme muss eingerichtet und betrieben werden."
|
||||
expected:
|
||||
routing_type: composite
|
||||
decompose_required: true
|
||||
|
||||
- id: GT-CONTAINER-005
|
||||
name: Secure development lifecycle is composite
|
||||
category: container_control_detection
|
||||
input: "Ein sicherer Softwareentwicklungsprozess ist umzusetzen."
|
||||
expected:
|
||||
routing_type: composite
|
||||
decompose_required: true
|
||||
|
||||
# E. Framework Decomposition (5)
|
||||
|
||||
- id: GT-FRAMEWORK-001
|
||||
name: CSA CCM AIS decomposes
|
||||
category: framework_decomposition
|
||||
input: "Die CCM-Praktiken für Application and Interface Security müssen implementiert werden."
|
||||
expected:
|
||||
routing_type: framework_container
|
||||
atomic_control_created_from_container_directly: false
|
||||
|
||||
- id: GT-FRAMEWORK-002
|
||||
name: OWASP ASVS V3 decomposes
|
||||
category: framework_decomposition
|
||||
input: "OWASP ASVS V3 Session Management Anforderungen sind umzusetzen."
|
||||
expected:
|
||||
routing_type: framework_container
|
||||
|
||||
- id: GT-FRAMEWORK-003
|
||||
name: NIST IA family decomposes
|
||||
category: framework_decomposition
|
||||
input: "NIST IA-Anforderungen zur Identifikation und Authentisierung sind umzusetzen."
|
||||
expected:
|
||||
routing_type: framework_container
|
||||
|
||||
- id: GT-FRAMEWORK-004
|
||||
name: OWASP API6 resource consumption decomposes
|
||||
category: framework_decomposition
|
||||
input: "OWASP API6 Anforderungen zu unrestricted resource consumption sind umzusetzen."
|
||||
expected:
|
||||
routing_type: framework_container
|
||||
|
||||
- id: GT-FRAMEWORK-005
|
||||
name: Framework domain not direct atomic
|
||||
category: framework_decomposition
|
||||
input: "Alle Controls der Kategorie Access Control müssen implementiert werden."
|
||||
expected:
|
||||
atomic_control_created_from_container_directly: false
|
||||
|
||||
# F. Evidence Leakage (5)
|
||||
|
||||
- id: GT-EVIDENCE-001
|
||||
name: SBOM evidence not duplicate control
|
||||
category: evidence_not_control
|
||||
input: "Ein SBOM-Nachweis muss vorliegen."
|
||||
expected:
|
||||
classification: evidence
|
||||
new_master_control_created: false
|
||||
|
||||
- id: GT-EVIDENCE-002
|
||||
name: Screenshot is evidence
|
||||
category: evidence_not_control
|
||||
input: "Ein Screenshot der MFA-Konfiguration ist vorzulegen."
|
||||
expected:
|
||||
classification: evidence
|
||||
new_master_control_created: false
|
||||
|
||||
- id: GT-EVIDENCE-003
|
||||
name: Audit report is evidence
|
||||
category: evidence_not_control
|
||||
input: "Ein Auditbericht zur Zugriffskontrolle muss vorhanden sein."
|
||||
expected:
|
||||
classification: evidence
|
||||
new_master_control_created: false
|
||||
|
||||
- id: GT-EVIDENCE-004
|
||||
name: Certification is evidence
|
||||
category: evidence_not_control
|
||||
input: "Eine Zertifizierung der Komponente ist nachzuweisen."
|
||||
expected:
|
||||
classification: evidence
|
||||
new_master_control_created: false
|
||||
|
||||
- id: GT-EVIDENCE-005
|
||||
name: Jira ticket export is evidence
|
||||
category: evidence_not_control
|
||||
input: "JIRA-Tickets zur Behebung nicht konformer Accounts müssen exportiert werden."
|
||||
expected:
|
||||
classification: evidence
|
||||
new_master_control_created: false
|
||||
|
||||
# G. Scope Dimension (5)
|
||||
|
||||
- id: GT-SCOPE-001
|
||||
name: Confidentiality actor scopes stay separate
|
||||
category: scope_dimension
|
||||
input:
|
||||
- "Mitarbeiter müssen Vertraulichkeit wahren."
|
||||
- "Unterauftragnehmer müssen Vertraulichkeit wahren."
|
||||
- "Externe Stellen müssen Vertraulichkeit wahren."
|
||||
expected:
|
||||
master_controls_count: 3
|
||||
|
||||
- id: GT-SCOPE-002
|
||||
name: Impartiality organization vs assessment
|
||||
category: scope_dimension
|
||||
input:
|
||||
- "Unparteilichkeit muss in Leitlinien verankert sein."
|
||||
- "Unparteilichkeit muss bei Konformitätsbewertungen sichergestellt sein."
|
||||
expected:
|
||||
master_controls_count: 2
|
||||
|
||||
- id: GT-SCOPE-003
|
||||
name: Provider size and AI complexity separate
|
||||
category: scope_dimension
|
||||
input:
|
||||
- "Verfahren müssen die Größe des Anbieters berücksichtigen."
|
||||
- "Verfahren müssen die Komplexität des KI-Systems berücksichtigen."
|
||||
expected:
|
||||
master_controls_count: 2
|
||||
|
||||
- id: GT-SCOPE-004
|
||||
name: Third-party and open-source components separate
|
||||
category: scope_dimension
|
||||
input:
|
||||
- "Drittkomponenten müssen geprüft werden."
|
||||
- "Open-Source-Komponenten müssen separat geprüft werden."
|
||||
expected:
|
||||
master_controls_count: 2
|
||||
|
||||
- id: GT-SCOPE-005
|
||||
name: Admin accounts and all users do not collapse
|
||||
category: scope_dimension
|
||||
input:
|
||||
- "Privilegierte Accounts müssen MFA verwenden."
|
||||
- "Alle Nutzer müssen MFA verwenden."
|
||||
expected:
|
||||
master_controls_count: 2
|
||||
|
||||
# H. Title Quality (5)
|
||||
|
||||
- id: GT-TITLE-001
|
||||
name: No truncated titles
|
||||
category: title_quality
|
||||
input: "Ressourcenkontingente für kostenpflichtige Drittanbieter-API-Integrationen definieren und überwachen."
|
||||
expected:
|
||||
truncated_title_allowed: false
|
||||
forbidden_title_endings: ["defin", "überw"]
|
||||
|
||||
- id: GT-TITLE-002
|
||||
name: Legal disclosure title safe
|
||||
category: title_quality
|
||||
input: "Informationen dürfen nur offengelegt werden, wenn dies gesetzlich erforderlich ist."
|
||||
expected:
|
||||
create: ["Gesetzlich erforderliche Offenlegung kontrolliert"]
|
||||
forbidden: ["Offenlegung erlangter Informationen"]
|
||||
|
||||
- id: GT-TITLE-003
|
||||
name: No generic implemented suffix for prohibitions
|
||||
category: title_quality
|
||||
input: "Session-IDs dürfen nicht in URLs oder Logs erscheinen."
|
||||
expected:
|
||||
create: ["Session-IDs in URLs oder Logs verhindert"]
|
||||
forbidden: ["Session-IDs in URLs oder Logs umgesetzt"]
|
||||
|
||||
- id: GT-TITLE-004
|
||||
name: German title normalization
|
||||
category: title_quality
|
||||
input: "maximale Payload-Größen für API-Anfragen definieren."
|
||||
expected:
|
||||
canonical_title: "Maximale Payload-Größen definiert"
|
||||
|
||||
- id: GT-TITLE-005
|
||||
name: No broad object as atomic title
|
||||
category: title_quality
|
||||
input: "Token-Schutz muss umgesetzt werden."
|
||||
expected:
|
||||
forbidden: ["Token-Schutz umgesetzt"]
|
||||
routing_type: composite
|
||||
Reference in New Issue
Block a user