compliance/api/tom_routes.py (609 LOC) -> 215 LOC thin routes +
434-line TOMService. Request bodies (TOMStateBody, TOMMeasureCreate,
TOMMeasureUpdate, TOMMeasureBulkItem, TOMMeasureBulkBody) moved to
compliance/schemas/tom.py (joining the existing response models from
the Step 3 split).
Single-service split (not two like banner): state, measures CRUD + bulk
upsert, stats, export, and version lookups are all tightly coupled
around the TOMMeasureDB aggregate, so splitting would create artificial
boundaries. TOMService is 434 LOC — comfortably under the 500 hard cap.
Domain error mapping:
- ConflictError -> 409 (version conflict on state save; duplicate control_id on create)
- NotFoundError -> 404 (missing measure on update; missing version)
- ValidationError -> 400 (missing tenant_id on DELETE /state)
Legacy test compat: the existing tests/test_tom_routes.py imports
TOMMeasureBulkItem, _parse_dt, _measure_to_dict, and DEFAULT_TENANT_ID
directly from compliance.api.tom_routes. All re-exported via __all__ so
the 44-test file runs unchanged.
mypy.ini flips compliance.api.tom_routes from ignore_errors=True to
False. TOMService carries the scoped Column[T] header.
Verified:
- 217/217 pytest (173 baseline + 44 TOM) pass
- OpenAPI 360/484 unchanged
- mypy compliance/ -> Success on 124 source files
- tom_routes.py 609 -> 215 LOC
- Hard-cap violations: 16 -> 15
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Phase 1 Step 4, file 2 of 18. Same cookbook as audit_routes (4a91814 +
883ef70) applied to banner_routes.py.
compliance/api/banner_routes.py (653 LOC) is decomposed into:
compliance/api/banner_routes.py (255) — thin handlers
compliance/services/banner_consent_service.py (298) — public SDK surface
compliance/services/banner_admin_service.py (238) — site/category/vendor CRUD
compliance/services/_banner_serializers.py ( 81) — ORM-to-dict helpers
shared between the
two services
compliance/schemas/banner.py ( 85) — Pydantic request models
Split rationale: the SDK-facing endpoints (consent CRUD, config
retrieval, export, stats) and the admin CRUD endpoints (sites +
categories + vendors) have distinct audiences and different auth stories,
and combined they would push the service file over the 500 hard cap.
Two focused services is cleaner than one ~540-line god class.
The shared ORM-to-dict helpers live in a private sibling module
(_banner_serializers) rather than a static method on either service, so
both services can import without a cycle.
Handlers follow the established pattern:
- Depends(get_consent_service) or Depends(get_admin_service)
- `with translate_domain_errors():` wrapping the service call
- Explicit return type annotations
- ~3-5 lines per handler
Services raise NotFoundError / ConflictError / ValidationError from
compliance.domain; no HTTPException in the service layer.
mypy.ini flips compliance.api.banner_routes from ignore_errors=True to
False, joining audit_routes in the strict scope. The services carry the
same scoped `# mypy: disable-error-code="arg-type,assignment"` header
used by the audit services for the ORM Column[T] issue.
Pydantic schemas moved to compliance.schemas.banner (mirroring the Step 3
schemas split). They were previously defined inline in banner_routes.py
and not referenced by anything outside it, so no backwards-compat shim
is needed.
Verified:
- 224/224 pytest (173 baseline + 26 audit integration + 25 banner
integration) pass
- tests/contracts/test_openapi_baseline.py green (360/484 unchanged)
- mypy compliance/ -> Success: no issues found in 123 source files
- All new files under the 300 soft target (largest: 298)
- banner_routes.py drops from 653 -> 255 LOC (below hard cap)
Hard-cap violations remaining: 16 (was 17).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Phase 1 Step 4 follow-up addressing the debt flagged in the worked-example
commit (4a91814).
## mypy --strict policy
Adds backend-compliance/mypy.ini declaring the strict-mode scope:
Fully strict (enforced today):
- compliance/domain/
- compliance/schemas/
- compliance/api/_http_errors.py
- compliance/api/audit_routes.py (refactored in Step 4)
- compliance/services/audit_session_service.py
- compliance/services/audit_signoff_service.py
Loose (ignore_errors=True) with a migration path:
- compliance/db/* — SQLAlchemy 1.x Column[] vs
runtime T; unblocks Phase 1
until a Mapped[T] migration.
- compliance/api/<route>.py — each route file flips to
strict as its own Step 4
refactor lands.
- compliance/services/<legacy util> — 14 utility services
(llm_provider, pdf_extractor,
seeder, ...) that predate the
clean-arch refactor.
- compliance/tests/ — excluded (legacy placeholder
style). The new TestClient-
based integration suite is
type-annotated.
The two new service files carry a scoped `# mypy: disable-error-code="arg-type,assignment"`
header for the ORM Column[T] issue — same underlying SQLAlchemy limitation,
narrowly scoped rather than wholesale ignore_errors.
Flow: `cd backend-compliance && mypy compliance/` -> clean on 119 files.
CI yaml updated to use the config instead of ad-hoc package lists.
## Bugs fixed while enabling strict
mypy --strict surfaced two latent bugs in the pre-refactor code. Both
were invisible because the old `compliance/tests/test_audit_routes.py`
is a placeholder suite that asserts on request-data shape and never
calls the handlers:
- AuditSessionResponse.updated_at is a required field in the schema,
but the original handler didn't pass it. Fixed in
AuditSessionService._to_response.
- PaginationMeta requires has_next + has_prev. The original audit
checklist handler didn't compute them. Fixed in
AuditSignOffService.get_checklist.
Both are behavior-preserving at the HTTP level because the old code
would have raised Pydantic ValidationError at response serialization
had the endpoint actually been exercised.
## Integration test suite
Adds backend-compliance/tests/test_audit_routes_integration.py — 26
real TestClient tests against an in-memory sqlite backend (StaticPool).
Replaces the coverage gap left by the placeholder suite.
Covers:
- Session CRUD + lifecycle transitions (draft -> in_progress -> completed
-> archived), including the 409 paths for illegal transitions
- Checklist pagination, filtering, search
- Sign-off create / update / auto-start-session / count-flipping
- Sign-off 400 (invalid result), 404 (missing requirement), 409 (completed session)
- Get-signoff 404 / 200 round-trip
Uses a module-scoped schema fixture + per-test DELETE-sweep so the
suite runs in ~2.3s despite the ~50-table ORM surface.
Verified:
- 199/199 pytest (173 original + 26 new audit integration) pass
- tests/contracts/test_openapi_baseline.py green, OpenAPI 360/484 unchanged
- mypy compliance/ -> Success: no issues found in 119 source files
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Phase 1 Step 4 of PHASE1_RUNBOOK.md, first worked example. Demonstrates
the router -> service delegation pattern for all 18 oversized route
files still above the 500 LOC hard cap.
compliance/api/audit_routes.py (637 LOC) is decomposed into:
compliance/api/audit_routes.py (198) — thin handlers
compliance/services/audit_session_service.py (259) — session lifecycle
compliance/services/audit_signoff_service.py (319) — checklist + sign-off
compliance/api/_http_errors.py ( 43) — reusable error translator
Handlers shrink to 3-6 lines each:
@router.post("/sessions", response_model=AuditSessionResponse)
async def create_audit_session(
request: CreateAuditSessionRequest,
service: AuditSessionService = Depends(get_audit_session_service),
):
with translate_domain_errors():
return service.create(request)
Services are HTTP-agnostic: they raise NotFoundError / ConflictError /
ValidationError from compliance.domain, and the route layer translates
those to HTTPException(404/409/400) via the translate_domain_errors()
context manager in compliance.api._http_errors. The error translator is
reusable by every future Step 4 refactor.
Services take a sqlalchemy Session in the constructor and are wired via
Depends factories (get_audit_session_service / get_audit_signoff_service).
No globals, no module-level state.
Behavior is byte-identical at the HTTP boundary:
- Same paths, methods, status codes, response models
- Same error messages (domain error __str__ preserved)
- Same auto-start-on-first-signoff, same statistics calculation,
same signature hash format, same PDF streaming response
Verified:
- 173/173 pytest compliance/tests/ tests/contracts/ pass
- OpenAPI 360 paths / 484 operations unchanged
- audit_routes.py under soft 300 target
- Both new service files under soft 300 / hard 500
Note: compliance/tests/test_audit_routes.py contains placeholder tests
that do not actually import or call the handler functions — they only
assert on request-data shape. Real behavioral coverage relies on the
contract test. A follow-up commit should add TestClient-based
integration tests for the audit endpoints. Flagged in PHASE1_RUNBOOK.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Squash of branch refactor/phase0-guardrails-and-models-split — 4 commits,
81 files, 173/173 pytest green, OpenAPI contract preserved (360 paths /
484 operations).
## Phase 0 — Architecture guardrails
Three defense-in-depth layers to keep the architecture rules enforced
regardless of who opens Claude Code in this repo:
1. .claude/settings.json PreToolUse hook on Write/Edit blocks any file
that would exceed the 500-line hard cap. Auto-loads in every Claude
session in this repo.
2. scripts/githooks/pre-commit (install via scripts/install-hooks.sh)
enforces the LOC cap locally, freezes migrations/ without
[migration-approved], and protects guardrail files without
[guardrail-change].
3. .gitea/workflows/ci.yaml gains loc-budget + guardrail-integrity +
sbom-scan (syft+grype) jobs, adds mypy --strict for the new Python
packages (compliance/{services,repositories,domain,schemas}), and
tsc --noEmit for admin-compliance + developer-portal.
Per-language conventions documented in AGENTS.python.md, AGENTS.go.md,
AGENTS.typescript.md at the repo root — layering, tooling, and explicit
"what you may NOT do" lists. Root CLAUDE.md is prepended with the six
non-negotiable rules. Each of the 10 services gets a README.md.
scripts/check-loc.sh enforces soft 300 / hard 500 and surfaces the
current baseline of 205 hard + 161 soft violations so Phases 1-4 can
drain it incrementally. CI gates only CHANGED files in PRs so the
legacy baseline does not block unrelated work.
## Deprecation sweep
47 files. Pydantic V1 regex= -> pattern= (2 sites), class Config ->
ConfigDict in source_policy_router.py (schemas.py intentionally skipped;
it is the Phase 1 Step 3 split target). datetime.utcnow() ->
datetime.now(timezone.utc) everywhere including SQLAlchemy default=
callables. All DB columns already declare timezone=True, so this is a
latent-bug fix at the Python side, not a schema change.
DeprecationWarning count dropped from 158 to 35.
## Phase 1 Step 1 — Contract test harness
tests/contracts/test_openapi_baseline.py diffs the live FastAPI /openapi.json
against tests/contracts/openapi.baseline.json on every test run. Fails on
removed paths, removed status codes, or new required request body fields.
Regenerate only via tests/contracts/regenerate_baseline.py after a
consumer-updated contract change. This is the safety harness for all
subsequent refactor commits.
## Phase 1 Step 2 — models.py split (1466 -> 85 LOC shim)
compliance/db/models.py is decomposed into seven sibling aggregate modules
following the existing repo pattern (dsr_models.py, vvt_models.py, ...):
regulation_models.py (134) — Regulation, Requirement
control_models.py (279) — Control, Mapping, Evidence, Risk
ai_system_models.py (141) — AISystem, AuditExport
service_module_models.py (176) — ServiceModule, ModuleRegulation, ModuleRisk
audit_session_models.py (177) — AuditSession, AuditSignOff
isms_governance_models.py (323) — ISMSScope, Context, Policy, Objective, SoA
isms_audit_models.py (468) — Finding, CAPA, MgmtReview, InternalAudit,
AuditTrail, Readiness
models.py becomes an 85-line re-export shim in dependency order so
existing imports continue to work unchanged. Schema is byte-identical:
__tablename__, column definitions, relationship strings, back_populates,
cascade directives all preserved.
All new sibling files are under the 500-line hard cap; largest is
isms_audit_models.py at 468. No file in compliance/db/ now exceeds
the hard cap.
## Phase 1 Step 3 — infrastructure only
backend-compliance/compliance/{schemas,domain,repositories}/ packages
are created as landing zones with docstrings. compliance/domain/
exports DomainError / NotFoundError / ConflictError / ValidationError /
PermissionError — the base classes services will use to raise
domain-level errors instead of HTTPException.
PHASE1_RUNBOOK.md at backend-compliance/PHASE1_RUNBOOK.md documents
the nine-step execution plan for Phase 1: snapshot baseline,
characterization tests, split models.py (this commit), split schemas.py
(next), extract services, extract repositories, mypy --strict, coverage.
## Verification
backend-compliance/.venv-phase1: uv python install 3.12 + pip -r requirements.txt
PYTHONPATH=. pytest compliance/tests/ tests/contracts/
-> 173 passed, 0 failed, 35 warnings, OpenAPI 360/484 unchanged
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The NOT EXISTS check and Duplicate Guard now exclude deprecated and
duplicate controls, enabling clean re-runs after invalidation.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Truncate object keys to 40 chars (was 80) at underscore boundary
- Strip German qualifying prepositional phrases (bei/für/gemäß/von/zur/...)
- Add 65 new synonym mappings for near-duplicate patterns found in analysis
- Strip trailing noise tokens (articles/prepositions)
- Add _truncate_at_boundary() helper and _QUALIFYING_PHRASE_RE regex
- 11 new tests for normalization improvements (227 total pass)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
1. Duplicate Guard: merge_hint-Lookup vor INSERT in _write_atomic_control()
verhindert semantisch identische Controls unter demselben Parent.
2. Severity-Kalibrierung: action_type-basiert statt blind vom Parent.
define/review/test → max medium, implement/monitor → max high.
3. Title-Truncation: Schnitt am Wortende statt mitten im Wort.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Die atomic_controls_dedup Collection (51k Punkte) enthaelt nur atomare
Controls ohne source_citation. Jetzt wird der Parent-Control aufgeloest,
der die Rechtsgrundlage traegt. Deduplizierung nach Parent-UUID verhindert
mehrfache Eintraege fuer die gleiche Regulation.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
qwen3.5 gibt Antworten im 'thinking'-Feld statt 'response' zurueck.
Mit think:false wird der Thinking-Mode deaktiviert und die Antwort
korrekt im response-Feld geliefert.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Ollama als eigener Enum-Wert neben self_hosted, damit die
docker-compose-Konfiguration (ollama) korrekt aufgeloest wird.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
SQLAlchemy's text() parser doesn't properly handle :param::type
syntax — it fails to recognize :dd as a bind parameter when followed
by ::jsonb. Using CAST(:dd AS jsonb) instead.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
SQLAlchemy sessions enter a failed state after SQL errors.
Without rollback(), all subsequent queries on the same session
fail with InFailedSqlTransaction. Added try/except with rollback
in _mark_duplicate, _mark_duplicate_to, _write_review, cross-group
pass, and the main phase1 loop.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
All Pass 0b controls have pattern_id=NULL. Rewritten to:
- Phase 1: Group by merge_group_hint (action:object:trigger), 52k groups
- Phase 2: Cross-group embedding search for semantically similar masters
- Qdrant search uses unfiltered cross-regulation endpoint
- API param changed: pattern_id → hint_filter
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Benchmark shows Haiku is 2.5x faster than Sonnet at 5x lower cost
for this JSON structuring task. Quality is equivalent.
$142 vs $705 for 75K obligations, ~2.8 days vs ~7 days.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Previous formula (batch_size * 1500) exceeded Claude's 16K output limit
for batch_size > 10, causing API failures and Ollama fallback.
New formula: min(16384, max(4096, batch_size * 500))
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- _write_atomic_control() now uses RETURNING id and inserts into
control_parent_links (M:N) with source_regulation, source_article,
and obligation_candidate_id parsed from parent's source_citation
- New _parse_citation() helper for JSONB source_citation extraction
- New GET /controls/{id}/traceability endpoint returning full chain:
parent links with obligations, child controls, source_count
- Backend: control_type filter (atomic/rich) for controls + count
- Frontend: Rechtsgrundlagen section in ControlDetail showing all
parent links per source regulation with obligation text + strength
- Frontend: Atomic/Rich filter dropdown in Control Library list
- Frontend: GenerationStrategyBadge recognizes 'pass0b' strategy
- Tests: 3 new tests for parent_link creation + citation parsing,
existing batch test mock updated for RETURNING clause
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add obligation refinement pipeline between Pass 0a and 0b:
- Merge pass: rule-based dedup of implementation-level duplicate obligations
within the same parent control (Jaccard similarity on action+object)
- Enrich pass: classify trigger_type (event/periodic/continuous) and detect
is_implementation_specific from obligation text (regex-based, no LLM)
- Pass 0b: skip merged obligations, cap severity for impl-specific, override
category to 'testing' for test obligations
- Migration 075: merged_into_id, trigger_type, is_implementation_specific
- Two new API endpoints: merge-obligations, enrich-obligations
- 30+ new tests (122 total, all passing)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Phase 1 (LLM Quality):
- Add format=json to all Ollama payloads (obligation_extractor, control_generator, citation_backfill)
- Add Chain-of-Thought analysis steps to Pass 0a/0b system prompts
Phase 2 (Retrieval Quality):
- Hybrid search via Qdrant Query API with RRF fusion + automatic text index (legal_rag.go)
- Fallback to dense-only search if Query API unavailable
- Cross-encoder re-ranking with BGE Reranker v2 (RERANK_ENABLED=false by default)
- CPU-only PyTorch dependency to keep Docker image small
Phase 3 (Data Layer):
- Cross-regulation dedup pass (threshold 0.95) links controls across regulations
- DedupResult.link_type field distinguishes dedup_merge vs cross_regulation
- Chunk size defaults updated 512/50 → 1024/128 for new ingestions only
- Existing collections and controls are NOT affected
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
eu_2023_1542 (Batterieverordnung), eu_2023_988 (GPSR), nist_sp800_218,
nist_privacy_1_0, owasp_mobile_top10 were defaulting to Rule 3 (restricted)
instead of their correct rules. This caused 68/71 controls to be flagged
as too_close in the last pipeline run.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add _detect_recital() to QA pipeline — flags controls where
source_original_text contains Erwägungsgrund markers instead of
article text (28% of controls with source text affected).
- Recital detection via regex + phrase matching in QA validation
- 10 new tests (TestRecitalDetection), 81 total
- ReviewCompare component for side-by-side duplicate comparison
- Review mode split: Duplikat-Verdacht vs Rule-3-ohne-Anchor tabs
- MkDocs: recital detection documentation
- Detection script for bulk analysis (scripts/find_recital_controls.py)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Updated both structure_batch and reformulate_batch prompts to return null
for chunks without actionable requirements (definitions, TOCs, scope-only).
Explicit instruction to always process annexes/appendices as they often
contain concrete technical requirements.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Local LLM prefilter (llama3.2 3B) was incorrectly skipping annex chunks
that contain concrete requirements. Added skip_prefilter flag to bypass
the local pre-filter and send all chunks directly to Anthropic API.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The embedding pre-load for 4998 existing controls takes ~16 minutes,
causing the SQLAlchemy session to become invalid. Added rollback after
pre-load completes to reset the session before subsequent DB operations.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Chunks without a regulation_code were silently passing through the filter
in _scan_rag(), causing unrelated documents (e.g. Data Act, legal templates)
to be included in filtered generation jobs. Now chunks without reg_code are
skipped when regulation_filter is active.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add Anthropic API support to decomposition Pass 0a/0b (prompt caching, content batching)
- Add Anthropic Batch API (50% cost reduction, async 24h processing)
- Add source_filter (ILIKE on source_citation) for regulation-based filtering
- Add category_filter to Pass 0a for selective decomposition
- Add regulation_filter to control_generator for RAG scan phase filtering
(prefix match on regulation_code — enables CE + Code Review focus)
- New API endpoints: batch-submit-0a, batch-submit-0b, batch-status, batch-process
- 83 new tests (all passing)
Cost reduction: $2,525 → ~$600-700 with all optimizations combined.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Implements Phases 1-4 of the IACE Hazard-Matching-Engine:
- 120 machine components (C001-C120) in 11 categories
- 20 energy sources (EN01-EN20)
- ~85 tag taxonomy across 5 domains
- 44 hazard patterns with AND/NOT matching logic
- Pattern engine with tag resolution and confidence scoring
- 8 new API endpoints (component-library, energy-sources, tags, patterns, match/apply)
- Completeness gate G09 for pattern matching
- 320 tests passing (36 new)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Group chunks by regulation_code before batching for better LLM context
- Add generation_strategy column (ungrouped=v1, document_grouped=v2)
- Add v1/v2 badge to control cards in frontend
- Add sort-by-source option with visual group headers
- Add frontend page tests (18 tests)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add 29 new document types (IT security, data, personnel, vendor, BCM
policies) to VALID_DOCUMENT_TYPES and 5 category pills to the document
generator UI. Include seed script for production DB population.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Many regulation codes (nist_sp800_53r5, eucsa, owasp_top10_2021, EDPB
guidelines, EU laws, AT/FR/ES/NL/IT/HU laws) were defaulting to Rule 3
(restricted) because they weren't in REGULATION_LICENSE_MAP. Now all
~100 regulation codes from RAG are properly mapped to Rule 1 or 2.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
from __future__ import annotations breaks Pydantic BaseModel runtime type
evaluation. Replaced str | None → Optional[str], list[str] → List[str] etc.
in control_generator.py, anchor_finder.py, control_generator_routes.py.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add legal context enrichment from Qdrant vector corpus to the two
highest-priority modules (Requirements AI assistant and DSFA drafting
engine).
Go SDK:
- Add SearchCollection() with collection override + whitelist validation
- Refactor Search() to delegate to shared searchInternal()
Python backend:
- New ComplianceRAGClient proxying POST /sdk/v1/rag/search (error-tolerant)
- AI assistant: enrich interpret_requirement() and suggest_controls() with RAG
- Requirements API: add ?include_legal_context=true query parameter
Admin (Next.js):
- Extract shared queryRAG() utility from chat route
- Inject RAG legal context into v1 and v2 draft pipelines
Tests for all three layers (Go, Python, TypeScript shared utility).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Sharang Parnerkar