Compare commits

...

98 Commits

Author SHA1 Message Date
Benjamin Admin
3904ddb493 fix(sub-columns): convert relative word positions to absolute coords for split
Some checks failed
CI / go-lint (push) Has been skipped
CI / python-lint (push) Has been skipped
CI / nodejs-lint (push) Has been skipped
CI / test-go-school (push) Successful in 24s
CI / test-go-edu-search (push) Successful in 27s
CI / test-python-klausur (push) Failing after 1m51s
CI / test-python-agent-core (push) Successful in 14s
CI / test-nodejs-website (push) Successful in 17s
Word 'left' values in ColumnGeometry.words are relative to the content
ROI (left_x), but geo.x is in absolute image coordinates. The split
position was computed from relative word positions and then compared
against absolute geo.x, resulting in negative widths and no splits on
real data. Pass left_x through to _detect_sub_columns to bridge the
two coordinate systems.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-02 19:16:13 +01:00
Benjamin Admin
6e1a349eed fix(tests): adjust word counts so 10% threshold works correctly
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-02 19:00:14 +01:00
Benjamin Admin
7252f9a956 refactor(ocr-pipeline): use left-edge alignment approach for sub-column detection
Replace gap-based splitting with alignment-bin approach: cluster word
left-edges within 8px tolerance, find the leftmost bin with >= 10% of
words as the true column start, split off any words to its left as a
sub-column. This correctly handles both page references ("p.59") and
misread exclamation marks ("!" → "I") even when the pixel gap is small.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-02 18:56:38 +01:00
Benjamin Admin
f13116345b fix(tests): use correct bbox_pct dict format in _cells_to_vocab_entries tests
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-02 18:26:24 +01:00
Benjamin Admin
991984d9c3 fix(tests): pass columns_meta arg to _cells_to_vocab_entries tests
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-02 18:23:55 +01:00
Benjamin Admin
1a246eb059 feat(ocr-pipeline): generic sub-column detection via left-edge clustering
Detects hidden sub-columns (e.g. page references like "p.59") within
already-recognized columns by clustering word left-edge positions and
splitting when a clear minority cluster exists. The sub-column is then
classified as page_ref and mapped to VocabRow.source_page.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-02 18:18:02 +01:00
Benjamin Admin
0532b2a797 fix(ocr-pipeline): skip edge-touching gaps in header/footer detection
Some checks failed
CI / go-lint (push) Has been skipped
CI / python-lint (push) Has been skipped
CI / nodejs-lint (push) Has been skipped
CI / test-go-school (push) Successful in 25s
CI / test-go-edu-search (push) Successful in 25s
CI / test-python-klausur (push) Failing after 1m50s
CI / test-python-agent-core (push) Successful in 15s
CI / test-nodejs-website (push) Successful in 16s
Gaps that extend to the image boundary (top/bottom edge) are not valid
content separators — they typically represent dewarp padding. Only gaps
with content on both sides qualify as header/footer boundaries.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-02 17:54:49 +01:00
Benjamin Admin
f1fcc67357 fix(ocr-pipeline): clamp gap detection to img_h to avoid dewarp padding
Some checks failed
CI / go-lint (push) Has been skipped
CI / python-lint (push) Has been skipped
CI / nodejs-lint (push) Has been skipped
CI / test-go-school (push) Successful in 26s
CI / test-go-edu-search (push) Successful in 27s
CI / test-python-klausur (push) Failing after 1m46s
CI / test-python-agent-core (push) Successful in 17s
CI / test-nodejs-website (push) Successful in 16s
The inverted image can be taller than img_h after dewarp shear
correction, causing footer_y to be detected outside the visible page.
Now clamps the horizontal projection to actual_h = min(inv.shape[0], img_h).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-02 17:06:58 +01:00
Benjamin Admin
c8981423d4 feat(ocr-pipeline): distinguish header/footer vs margin_top/margin_bottom
Some checks failed
CI / go-lint (push) Has been skipped
CI / python-lint (push) Has been skipped
CI / nodejs-lint (push) Has been skipped
CI / test-go-school (push) Successful in 29s
CI / test-go-edu-search (push) Successful in 27s
CI / test-python-klausur (push) Failing after 2m0s
CI / test-python-agent-core (push) Successful in 18s
CI / test-nodejs-website (push) Successful in 19s
Check for actual ink content in detected top/bottom regions:
- 'header'/'footer' when text is present (e.g. title, page number)
- 'margin_top'/'margin_bottom' when the region is empty page margin

Also update all skip-type sets and color maps for the new types.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-02 16:55:41 +01:00
Benjamin Admin
f615c5f66d feat(ocr-pipeline): generic header/footer detection via projection gap analysis
Some checks failed
CI / go-lint (push) Has been skipped
CI / python-lint (push) Has been skipped
CI / nodejs-lint (push) Has been skipped
CI / test-go-school (push) Successful in 25s
CI / test-go-edu-search (push) Successful in 25s
CI / test-python-klausur (push) Failing after 1m48s
CI / test-python-agent-core (push) Successful in 17s
CI / test-nodejs-website (push) Successful in 16s
Replace the trivial top_y/bottom_y threshold check with horizontal
projection gap analysis that finds large whitespace gaps separating
header/footer content from the main body. This correctly detects
headers (e.g. "VOCABULARY" banners) and footers (page numbers) even
when _find_content_bounds includes them in the content area.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-02 16:13:48 +01:00
Benjamin Admin
a052f73de3 fix(ocr-pipeline): pass left_x/right_x to classify_column_types in API path
Some checks failed
CI / go-lint (push) Has been skipped
CI / python-lint (push) Has been skipped
CI / nodejs-lint (push) Has been skipped
CI / test-go-school (push) Successful in 25s
CI / test-go-edu-search (push) Successful in 26s
CI / test-python-klausur (push) Failing after 1m45s
CI / test-python-agent-core (push) Successful in 15s
CI / test-nodejs-website (push) Successful in 18s
The ocr_pipeline_api.py code path called classify_column_types without
left_x/right_x, so margin regions were never created. Also add logging
to _build_margin_regions for debugging.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-02 15:42:39 +01:00
Benjamin Admin
34ccdd5fd1 feat(ocr-pipeline): filter scan artifacts in content bounds and add margin regions
Some checks failed
CI / go-lint (push) Has been skipped
CI / python-lint (push) Has been skipped
CI / nodejs-lint (push) Has been skipped
CI / test-go-school (push) Successful in 26s
CI / test-go-edu-search (push) Successful in 27s
CI / test-python-klausur (push) Failing after 1m50s
CI / test-python-agent-core (push) Successful in 16s
CI / test-nodejs-website (push) Successful in 18s
Thin black lines (1-5px) at page edges from scanning were incorrectly
detected as content, shifting content bounds and creating spurious
IGNORE columns. This filters narrow projection runs (<1% of image
dimension) and introduces explicit margin_left/margin_right regions
for downstream page reconstruction.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-02 15:29:18 +01:00
Benjamin Admin
e718353d9f feat(ocr-pipeline): 6 systematic improvements for robustness, performance & UX
Some checks failed
CI / go-lint (push) Has been skipped
CI / python-lint (push) Has been skipped
CI / nodejs-lint (push) Has been skipped
CI / test-go-school (push) Successful in 37s
CI / test-go-edu-search (push) Successful in 26s
CI / test-python-klausur (push) Failing after 1m57s
CI / test-python-agent-core (push) Successful in 19s
CI / test-nodejs-website (push) Successful in 21s
1. Unit tests: 76 new parametrized tests for noise filter, phonetic detection,
   cell text cleaning, and row merging (116 total, all green)
2. Continuation-row merge: detect multi-line vocab entries where text wraps
   (lowercase EN + empty DE) and merge into previous entry
3. Empty DE fallback: secondary PSM=7 OCR pass for cells missed by PSM=6
4. Batch-OCR: collect empty cells per column, run single Tesseract call on
   column strip instead of per-cell (~66% fewer calls for 3+ empty cells)
5. StepReconstruction UI: font scaling via naturalHeight, empty EN/DE field
   highlighting, undo/redo (Ctrl+Z), per-cell reset button
6. Session reprocess: POST /sessions/{id}/reprocess endpoint to re-run from
   any step, with reprocess button on completed pipeline steps

Also fixes pre-existing dewarp_image tuple unpacking bug in run_cv_pipeline
and updates dewarp tests to match current (image, info) return signature.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 14:46:38 +01:00
Benjamin Admin
c3a924a620 fix(ocr-pipeline): merge phonetic-only rows and fix bracket noise filter
Two fixes:
1. Tokens ending with ] (e.g. "serva]") were stripped by the noise
   filter because ] was not in the allowed punctuation list.
2. Rows containing only phonetic transcription (e.g. ['mani serva])
   are now merged into the previous vocab entry instead of creating
   a separate (invalid) entry. This prevents the LLM from trying
   to "correct" phonetic fragments.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 14:14:20 +01:00
Benjamin Admin
650f15bc1b fix(ocr-pipeline): tolerate dictionary punctuation in noise filter
The noise filter was stripping words containing hyphens, parentheses,
slashes, and dots (e.g. "money-saver", "Schild(chen)", "(Salat-)Gurke",
"Tanz(veranstaltung)"). Now strips all common dictionary punctuation
before checking for internal noise characters.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 13:12:40 +01:00
Benjamin Admin
40a77a82f6 fix(ocr-pipeline): use midpoint boundaries for column word assignment
Replace containment-with-padding approach with midpoint-based column
ranges. For adjacent columns, the assignment boundary is the midpoint
between them (Voronoi-style). This prevents padding overlap where words
near column borders (e.g. "We" at the start of example sentences) were
assigned to the preceding column. The last column extends generously to
capture all rightmost text.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 12:53:56 +01:00
Benjamin Admin
87931c35e4 fix(ocr-pipeline): stop noise filter from stripping parenthesized words
_is_noise_tail_token() treated words with unbalanced parentheses like
"selbst)" or "(wir" as OCR noise because the parenthesis counted as
"internal noise". Now strips leading/trailing parentheses before the
noise check, so legitimate words in example sentences like
"We baked ... (wir ... selbst)" are preserved.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 12:51:28 +01:00
Benjamin Admin
29b1d95acc fix(ocr-pipeline): improve word-column assignment and LLM review accuracy
Word assignment: Replace nearest-center-distance with containment-first
strategy. Words whose center falls within a column's bounds (+ 15% pad)
are assigned to that column before falling back to nearest-center. This
fixes long example sentences losing their rightmost words to adjacent
columns.

LLM review: Strengthen prompt to explicitly forbid changing proper nouns,
place names, and correctly-spelled words. Add _is_spurious_change()
post-filter that rejects case-only changes and hallucinated word
replacements (< 50% character overlap).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 12:40:26 +01:00
Benjamin Admin
dbf0db0c13 feat(ocr-pipeline): improve LLM review UI + add reconstruction step
StepLlmReview: Show full vocab table with image overlay, row-level
status tracking (pending/active/reviewed/corrected/skipped), and
auto-scroll during SSE streaming. Load previous results on mount.

StepReconstruction: New step 7 with editable text fields at original
bbox positions over dewarped image. Zoom controls, tab navigation,
color-coded columns, save to backend.

Backend: Add POST /sessions/{id}/reconstruction endpoint.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 12:19:21 +01:00
Benjamin Admin
2a493890b6 feat(ocr-pipeline): add SSE streaming and phonetic filter to LLM review
- Stream LLM review results batch-by-batch (8 entries per batch) via SSE
- Frontend shows live progress bar, batch log, and corrections appearing
- Skip entries with IPA phonetic transcriptions (already dictionary-corrected)
- Refactor llm_review_entries into reusable helpers for both streaming and non-streaming paths

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 11:46:06 +01:00
Benjamin Admin
e171a736e7 fix(ocr-pipeline): increase LLM timeout to 300s and disable qwen3 thinking
- Add /no_think tag to prompt (qwen3 thinking mode causes massive slowdown)
- Increase httpx timeout from 120s to 300s for large vocab tables
- Improve error logging with traceback and exception type

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 11:31:03 +01:00
Benjamin Admin
938d1d69cf feat(ocr-pipeline): add LLM-based OCR correction step (Step 6)
Replace the placeholder "Koordinaten" step with an LLM review step that
sends vocab entries to qwen3:30b-a3b via Ollama for OCR error correction
(e.g. "8en" → "Ben"). Teachers can review, accept/reject individual
corrections in a diff table before applying them.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 11:13:17 +01:00
Benjamin Admin
e9f368d3ec feat(ocr-pipeline): add abbreviation allowlist to noise filter
Add _KNOWN_ABBREVIATIONS set with ~150 common EN/DE abbreviations
(sth, sb, etc, eg, ie, usw, bzw, vgl, adj, adv, prep, sg, pl, ...).
Tokens matching known abbreviations are never stripped as noise.

Also handle dotted abbreviations (e.g., z.B., i.e.) that have no
2+ consecutive alpha chars by checking the abbreviation set before
the _RE_REAL_WORD filter.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 10:46:54 +01:00
Benjamin Admin
3028f421b4 feat(ocr-pipeline): add cell text noise filter for OCR artifacts
Add _clean_cell_text() with three sub-filters to remove OCR noise:
- _is_garbage_text(): vowel/consonant ratio check for phantom row garbage
- _is_noise_tail_token(): dictionary-based trailing noise detection
- _RE_REAL_WORD check for cells with no real words (just fragments)

Handles balanced parentheses "(auf)" and trailing hyphens "under-"
as legitimate tokens while stripping noise like "Es)", "3", "ee", "B".

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 10:19:31 +01:00
Benjamin Admin
2b1c499d54 fix(ocr-pipeline): filter OCR noise from image areas and artifacts
Two generic noise filters added to _ocr_single_cell():

1. Word confidence filter (conf < 30): removes low-confidence words
   before text assembly.  Catches trailing artifacts like "Es)" after
   real text, and standalone noise from image edges.

2. Cell noise filter: clears cells whose entire text has no real
   alphabetic word (>= 2 letters).  Catches fragments like "E:", "3",
   "u", "D", "2.77", "and )" from image areas, while keeping real
   short words like "Ei", "go", "an".

Both filters apply to word-lookup AND cell-OCR fallback results.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 09:56:54 +01:00
Benjamin Admin
72cc77dcf4 fix(ocr-pipeline): cells = result, no post-processing content shuffling
The cell grid IS the result. Each cell stays at its detected position.

Removed _split_comma_entries and _attach_example_sentences from the
pipeline — they were shuffling content between rows/columns, causing
"Mäuse" to appear in a separate row, "stand..." to move to Example,
and "Ei" to disappear.

Now: cells → _cells_to_vocab_entries (1:1 row mapping) →
_fix_character_confusion → _fix_phonetic_brackets → done.

Also lowered pixel-density threshold from 2% to 0.5% for the cell-OCR
fallback so small text like "Ei" is not filtered out.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 09:41:30 +01:00
Benjamin Admin
e3f939a628 refactor(ocr-pipeline): make post-processing fully generic
Three non-generic solutions replaced with universal heuristics:

1. Cell-OCR fallback: instead of restricting to column_en/column_de,
   now checks pixel density (>2% dark pixels) for ANY column type.
   Truly empty cells are skipped without running Tesseract.

2. Example-sentence detection: instead of checking for example-column
   text (worksheet-specific), now uses sentence heuristics (>=4 words
   or ends with sentence punctuation). Short EN text without DE is
   kept as a vocab entry (OCR may have missed the translation).

3. Comma-split: re-enabled with singular/plural detection. Pairs like
   "mouse, mice" / "Maus, Mäuse" are kept together. Verb forms like
   "break, broke, broken" are still split into individual entries.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 09:27:30 +01:00
Benjamin Admin
6bca3370e0 fix(ocr-pipeline): fix vocab post-processing destroying correct cell results
Three bugs in the post-processing pipeline were overwriting correct
streaming results with wrong ones:

1. _split_comma_entries was splitting "Maus, Mäuse" into two separate
   entries. Disabled — word forms belong together.

2. _attach_example_sentences treated "Ei" (2 chars) as OCR noise due
   to `len(de) > 2` threshold. Lowered to `len(de) > 1`.

3. _attach_example_sentences wrongly classified rows with EN text but
   no DE (like "stand ...") as example sentences, merging them into
   the previous entry. Now only treats rows as examples if they also
   have no text in the example column.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 09:16:50 +01:00
Benjamin Admin
befc44d2dd perf(ocr-pipeline): limit cell-OCR fallback to EN/DE columns only
Skip Tesseract fallback for column_example cells which are often
legitimately empty.  This reduces ~48 Tesseract calls to ~10,
cutting Step 5 fallback time from ~13s to ~3s.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 09:01:08 +01:00
Benjamin Admin
6db3c02db4 fix(admin-lehrer): force unique build ID to bust browser caches
Next.js was producing the same chunk hash across builds, causing
browsers to serve stale cached JS even after redeployment.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 08:54:05 +01:00
Benjamin Admin
8f2c2e8f68 feat(ocr-pipeline): hybrid word-lookup with cell-OCR fallback
Word-lookup from full-page Tesseract is fast but can miss small or
isolated words (e.g. "Ei"). Now falls back to per-cell Tesseract OCR
for cells that remain empty after word-lookup. The ocr_engine field
reports 'cell_ocr_fallback' for cells that needed the fallback.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 08:21:12 +01:00
Benjamin Admin
50ad06f43a fix(ocr-pipeline): always run fresh word detection, skip stale cache
Word-lookup is now ~0.03s (vs seconds with per-cell Tesseract), so
always re-run detection when entering Step 5 instead of showing
potentially stale cached word_result from the session DB.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 08:05:13 +01:00
Benjamin Admin
2c4160e4c4 fix(ocr-pipeline): exclusive word-to-column assignment prevents duplicates
Replace per-cell word filtering (which allowed the same word to appear in
multiple columns due to padded overlap) with exclusive nearest-center
assignment. Each word is assigned to exactly one column per row.

Also use row height as Y-tolerance for text assembly so words within
the same row (e.g. "Maus, Mäuse") are always grouped on one line.

Fixes: words leaking into wrong columns, missing words, duplicate words.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 07:54:45 +01:00
Benjamin Admin
9bbde1c03e fix(ocr-pipeline): re-populate row.words for word-lookup in Step 5
The row_result stored in DB excludes words to keep payload small.
When Step 5 reconstructs RowGeometry from DB, words were empty,
causing word-lookup to find nothing and return blank cells.

Now re-populates row.words from cached _word_dicts (or re-runs
detect_column_geometry if cache is cold) before cell grid building.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 07:38:33 +01:00
Benjamin Admin
77869e32f4 feat(ocr-pipeline): use word-lookup instead of cell-OCR for cell grid
Replace per-cell Tesseract re-runs with lookup of pre-existing full-page
words from row.words. Words are filtered by X-overlap with column bounds.
This fixes phantom rows with garbage text, missing last words, and
incomplete example text by using the more reliable full-page OCR results.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 07:24:46 +01:00
Benjamin Admin
89b5f49918 fix(ocr-pipeline): filter phantom rows with word_count=0 from cell grid
Rows in inter-line whitespace gaps have no Tesseract words assigned but
were still processed by build_cell_grid, producing garbage OCR output.
Filter these phantom rows using the word_count field set during Step 4.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-01 18:40:13 +01:00
Benjamin Admin
7f27783008 feat(ocr-pipeline): add SSE streaming for word recognition (Step 5)
Cells now appear one-by-one in the UI as they are OCR'd, with a live
progress bar, instead of waiting for the full result.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-01 17:54:20 +01:00
Benjamin Admin
a666e883da fix(ocr-pipeline): exclude header/footer/page_ref from cell grid columns
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-01 17:33:48 +01:00
Benjamin Admin
27b895a848 feat(ocr-pipeline): generic cell-grid with optional vocab mapping
Extract build_cell_grid() as layout-agnostic foundation from
build_word_grid(). Step 5 now produces a generic cell grid (columns x
rows) and auto-detects whether vocab layout is present. Frontend
dynamically switches between vocab table (EN/DE/Example) and generic
cell table based on layout type.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-01 17:22:56 +01:00
Benjamin Admin
3bcb7aa638 fix(ocr-pipeline): remove overzealous grid row count validation
The validation that rejected word-center grid when it produced more rows
than gap-based detection was causing fallback to gap-based rows (large
boxes). The word-center grid regularization works correctly after the
center-based grouping and cluster merging fixes.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-01 13:01:27 +01:00
Benjamin Admin
c4f2e6554e fix(ocr-pipeline): prevent grid from producing more rows than gap-based
Two fixes:
1. Grid validation: reject word-center grid if it produces MORE rows
   than gap-based detection (more rows = lines were split = worse).
   Falls back to gap-based rows in that case.

2. Words overlay: draw clean grid cells (column × row intersections)
   instead of padded entry bboxes. Eliminates confusing double lines.
   OCR text labels are placed inside the grid cells directly.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-01 12:52:41 +01:00
Benjamin Admin
8e861e5a4d fix(ocr-pipeline): use gap-based row height for cluster tolerance
The y_tolerance for word-center clustering was based on median word
height (21px → 12px tolerance), which was too small. Words on the
same line can have centers 15-20px apart due to different heights.

Now uses 40% of the gap-based median row height as tolerance (e.g.
40px row → 16px tolerance), and 30% for merge threshold. This
produces correct cluster counts matching actual text lines.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-01 12:34:15 +01:00
Benjamin Admin
4970ca903e fix(ocr-pipeline): invalidate downstream results when steps are re-run
When columns change (Step 3), invalidate row_result and word_result.
When rows change (Step 4), invalidate word_result.
This ensures Step 5 always uses the latest row boundaries instead of
showing stale cached word_result from a previous run.

Applies to both auto-detection and manual override endpoints.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-01 12:24:44 +01:00
Benjamin Admin
97d4355aa9 fix(ocr-pipeline): group words by vertical center, merge close clusters
Fix half-height rows caused by tall special characters (brackets, IPA
symbols) being split into separate line clusters:

- Group words by vertical CENTER instead of TOP position, so tall
  characters on the same line stay in one cluster
- Filter outlier-height words (>2× median) when computing letter_h
  so brackets/IPA don't skew the row height
- Merge clusters closer than 0.4× median word height (definitely
  same text line despite slight center differences)
- Increased y_tolerance from 0.5× to 0.6× median word height
- Enhanced logging with cluster merge count and row height range

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-01 12:14:42 +01:00
Benjamin Admin
8ad5823fd8 feat(ocr-pipeline): word-center grid with section-break detection
Replace rigid uniform grid with bottom-up approach that derives row
boundaries from word vertical centers:
- Group words into line clusters, compute center_y per cluster
- Compute pitch (distance between consecutive centers)
- Detect section breaks where gap > 1.8× median pitch
- Place row boundaries at midpoints between consecutive centers
- Per-section local pitch adapts to heading/paragraph spacing
- Validate ≥85% word placement, fallback to gap-based rows

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-01 12:04:08 +01:00
Benjamin Admin
ec47045c15 feat(ocr-pipeline): uniform grid regularization for row detection (Step 7)
Replace _split_oversized_rows() with _regularize_row_grid(). When ≥60%
of content rows have consistent height (±25% of median), overlay a
uniform grid with the standard row height over the entire content area.
This leverages the fact that books/vocab lists use constant row heights.

Validates grid by checking ≥85% of words land in a grid row. Falls back
to gap-based rows if heights are too irregular or words don't fit.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-01 11:50:50 +01:00
Benjamin Admin
ba65e47654 feat(ocr-pipeline): move oversized row splitting from Step 5 to Step 4
Implement _split_oversized_rows() in detect_row_geometry() (Step 7) to
split content rows >1.5× median height using local horizontal projection.
This produces correctly-sized rows before word OCR runs, instead of
working around the issue in Step 5 with sub-cell splitting hacks.

Removed Step 5 workarounds: _split_oversized_entries(), sub-cell
splitting in build_word_grid(), and median_row_h calculation.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-01 11:46:18 +01:00
Benjamin Admin
8507e2e035 fix(ocr-pipeline): split oversized cells before OCR to capture all text
For cells taller than 1.5× median row height, split vertically into
sub-cells and OCR each separately. This fixes RapidOCR losing text
at the bottom of tall cells (e.g. "floor/Fußboden" below "egg/Ei"
in a merged row). Generic fix — works for any oversized cell.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-01 11:32:10 +01:00
Benjamin Admin
854d8b431b feat(rag-qa): add 14 missing PDF mappings for EDPB, ENISA, EDPS, TMG, UrhG
Adds entries for all regulation codes in REGULATIONS_IN_RAG that were
missing from RAG_PDF_MAPPING, fixing "Kein PDF-Mapping" messages.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-01 11:10:09 +01:00
Benjamin Admin
f2521d2b9e feat(ocr-pipeline): British/American IPA pronunciation choice
- Integrate Britfone dictionary (MIT, 15k British English IPA entries)
- Add pronunciation parameter: 'british' (default) or 'american'
- British uses Britfone (Received Pronunciation), falls back to CMU
- American uses eng_to_ipa/CMU, falls back to Britfone
- Frontend: dropdown to switch pronunciation, default = British
- API: ?pronunciation=british|american query parameter

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-01 11:08:52 +01:00
Benjamin Admin
954d21e469 fix: use local Inter font to avoid Google Fonts timeout in Docker build
The Docker container cannot reach Google Fonts, causing build failures.
Switch to bundled local font file using next/font/local.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-28 21:26:34 +01:00
Benjamin Admin
010616be5a fix(ocr-pipeline): generic example attachment + cell padding
1. Semantic example matching: instead of attaching example sentences
   to the immediately preceding entry, find the vocab entry whose
   English word(s) appear in the example. "a broken arm" → matches
   "broken" via word overlap, not "egg/Ei". Uses stem matching for
   word form variants (break/broken share stem "bro").

2. Cell padding: add 8px padding to each cell region so words at
   column/row edges don't get clipped by OCR (fixes "er wollte"
   missing at cell boundaries).

3. Treat very short DE text (≤2 chars) as OCR noise, not real
   translation — prevents false positives in example detection.

All fixes are generic and deterministic.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-28 21:24:28 +01:00
Benjamin Admin
e3aa8e899e feat(rag-qa): add fullscreen mode for split-view chunk browser
Allows viewing chunks side-by-side with original PDF in fullscreen mode
for large screen QA review. Toggle via button or close with Escape key.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-28 21:23:32 +01:00
Benjamin Admin
266b9dfad3 Fix PDF 404: default to bp_compliance_ce collection, add PDF existence check
Default collection changed from bp_compliance_gesetze (DE/AT/CH laws where
PDFs need manual download) to bp_compliance_ce (EU regulations where PDFs
are auto-downloaded). Added HEAD request check so missing PDFs show a clear
"PDF nicht vorhanden" message instead of a 404 in the iframe.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-28 21:13:26 +01:00
Benjamin Admin
ab294d5a6f feat(ocr-pipeline): deterministic post-processing pipeline
Add 4 post-processing steps after OCR (no LLM needed):

1. Character confusion fix: I/1/l/| correction using cross-language
   context (if DE has "Ich", EN "1" → "I")
2. IPA dictionary replacement: detect [phonetics] brackets, look up
   correct IPA from eng_to_ipa (MIT, 134k words) — replaces OCR'd
   phonetic symbols with dictionary-correct transcription
3. Comma-split: "break, broke, broken" / "brechen, brach, gebrochen"
   → 3 individual entries when part counts match
4. Example sentence attachment: rows with EN but no DE translation
   get attached as examples to the preceding vocab entry

All fixes are deterministic and generic — no hardcoded word lists.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-28 21:00:09 +01:00
Benjamin Admin
b48cd8bb46 Fix ChunkBrowserQA layout: proper height constraints, remove bottom nav duplication
- Root container uses calc(100vh - 220px) for fixed viewport height
- All flex children use min-h-0 to enable proper overflow scrolling
- Removed duplicate bottom nav buttons (Zurueck/Weiter) that appeared
  in the middle of the chunk text — navigation is only in the header now
- Chunk text panel scrolls internally with fixed header
- Added prominent article/section badges in header and panel header
- Added chunk length quality indicator (warns on very short/long chunks)
- Structural metadata keys (article, section, pages) sorted first
- Sidebar shows regulation name instead of code for better readability
- PDF viewer uses pages metadata from payload when available

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-28 20:24:50 +01:00
Benjamin Admin
d481e0087b deps: add eng-to-ipa for IPA dictionary lookup
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-28 20:23:40 +01:00
Benjamin Admin
f7e0f2bb4f feat(ocr-pipeline): line breaks, hyphen rejoin & oversized row splitting
- Preserve \n between visual lines within cells (instead of joining with space)
- Rejoin hyphenated words split across line breaks (e.g. Fuß-\nboden → Fußboden)
- Split oversized rows (>1.5× median height) into sub-entries when EN/DE
  line counts match — deterministic fix for missed Step 4 row boundaries
- Frontend: render \n as <br/>, use textarea for multiline editing

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-28 18:49:28 +01:00
Benjamin Admin
e7fb9d59f1 Fix ChunkBrowserQA: use regulation_id from Qdrant payload instead of regulation_code
The Qdrant collections use regulation_id (e.g. eu_2016_679) as the filter key,
not regulation_code (e.g. GDPR). Updated rag-constants.ts with correct qdrant_id
mappings from actual Qdrant data, fixed API to filter on regulation_id, and updated
ChunkBrowserQA to pass qdrant_id values.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-28 18:22:12 +01:00
Benjamin Admin
859342300e fix(ocr-pipeline): configure RapidOCR for German + tighter word detection
- Switch to PP-OCRv5 Latin model (supports ä, ö, ü, ß)
- Use SERVER model for better accuracy
- Lower Det.unclip_ratio 1.6→1.3 to reduce word merging
- Raise Det.box_thresh 0.5→0.6 for stricter detection

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-28 18:17:49 +01:00
Benjamin Admin
8c42fefa77 feat(rag): add QA Split-View Chunk-Browser for ingestion verification
New ChunkBrowserQA component replaces inline chunk browser with:
- Document sidebar with live chunk counts per regulation (batched Qdrant count API)
- Sequential chunk navigation with arrow keys (1/N through all chunks of a document)
- Overlap display showing previous/next chunk boundaries (amber-highlighted)
- Split-view with original PDF via iframe (estimated page from chunk index)
- Adjustable chunks-per-page ratio for PDF page estimation

Extracts REGULATIONS_IN_RAG and REGULATION_INFO to shared rag-constants.ts.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-28 17:46:11 +01:00
Benjamin Admin
984dfab975 fix(ocr-pipeline): add libgl1 for RapidOCR OpenCV dependency
RapidOCR pulls in full opencv-python which requires libGL.so.1

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-28 17:30:12 +01:00
Benjamin Admin
45435f226f feat(ocr-pipeline): line grouping fix + RapidOCR integration
Fix A: Use _group_words_into_lines() with adaptive Y-tolerance to
correctly order words in multi-line cells (fixes word reordering bug).

RapidOCR: Add as alternative OCR engine (PaddleOCR models on ONNX
Runtime, native ARM64). Engine selectable via dropdown in UI or
?engine= query param. Auto mode prefers RapidOCR when available.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-28 17:13:58 +01:00
Benjamin Admin
4ec7c20490 feat(ocr-pipeline): add rapidocr + onnxruntime to requirements
RapidOCR uses PaddleOCR models on ONNX Runtime, works natively on ARM64.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-28 17:08:21 +01:00
Benjamin Admin
17604b8eb2 test: add tests for API proxy scroll/collection-count and Chunk-Browser logic
Some checks failed
CI / go-lint (push) Has been skipped
CI / python-lint (push) Has been skipped
CI / nodejs-lint (push) Has been skipped
CI / test-go-school (push) Successful in 27s
CI / test-go-edu-search (push) Successful in 25s
CI / test-python-klausur (push) Failing after 1m41s
CI / test-python-agent-core (push) Successful in 14s
CI / test-nodejs-website (push) Successful in 19s
42 tests covering:
- Qdrant scroll endpoint proxy (offset, limit, filters, text search)
- Collection-count endpoint
- REGULATION_SOURCES URL validation (IFRS, EFRAG, ENISA, NIST, OECD)
- Chunk-Browser collections, text search filtering, pagination state

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-28 16:46:42 +01:00
Benjamin Admin
f39314fb27 docs: add Chunk-Browser documentation
- Document Chunk-Browser tab functionality and API
- Cover scroll endpoint, text search, pagination
- Document Originalquelle links and low-chunk warnings

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-28 09:50:36 +01:00
Benjamin Admin
356d39d6ee fix(ocr-pipeline): use PSM 6 (block) for multi-line cell OCR in word grid
PSM 7 (single line) missed the second line in cells with two lines.
PSM 6 handles multi-line content. Also fix sort order to Y-then-X
for correct reading order.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-28 09:40:04 +01:00
Benjamin Admin
491df4e1b0 feat: add Chunk-Browser tab to RAG page
- New 'Chunk-Browser' tab for sequential chunk browsing
- Qdrant scroll API proxy (scroll + collection-count actions)
- Pagination with prev/next through all chunks in a collection
- Text search filter with highlighting
- Click to expand chunk and see all metadata
- 'In Chunks suchen' button now navigates to Chunk-Browser with correct collection

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-28 09:35:52 +01:00
Benjamin Admin
954103cdf2 feat(ocr-pipeline): add Step 5 word recognition (grid from columns × rows)
Backend: build_word_grid() intersects column regions with content rows,
OCRs each cell with language-specific Tesseract, and returns vocabulary
entries with percent-based bounding boxes. New endpoints: POST /words,
GET /image/words-overlay, ground-truth save/retrieve for words.
Frontend: StepWordRecognition with overview + step-through labeling modes,
goToStep callback for row correction feedback loop.
MkDocs: OCR Pipeline documentation added.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-28 02:18:29 +01:00
Benjamin Admin
47dc2e6f7a feat(rag): source URLs, low-chunk warnings & IFRS/EFRAG entries
- Add REGULATION_SOURCES map with 88 original document URLs for all
  regulations (EUR-Lex, gesetze-im-internet.de, RIS, Fedlex, etc.)
- Render "Originalquelle →" link in regulation detail panel
- Add amber warning indicator for suspiciously low chunk counts (<10)
- Add EU_IFRS_DE, EU_IFRS_EN, EFRAG_ENDORSEMENT to RAG tracking

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-28 01:56:09 +01:00
Benjamin Admin
203b3c0e2d fix(ocr-pipeline): mask out images in row detection horizontal projection
Build a word-coverage mask so only pixels near Tesseract word bounding
boxes contribute to the horizontal projection. Image regions (high ink
but no words) are treated as white, preventing illustrations from
merging multiple vocabulary rows into one.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-28 01:39:20 +01:00
Benjamin Admin
b58aecd081 feat(ocr-pipeline): add Step 4 row detection UI in admin frontend
Insert rows step between columns and words in the pipeline wizard.
Shows overlay image, row list with type badges, and ground truth controls.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-28 01:28:05 +01:00
Benjamin Admin
04b83d5f46 feat(ocr-pipeline): add row detection step with horizontal gap analysis
Add Step 4 (row detection) between column detection and word recognition.
Uses horizontal projection profiles + whitespace gaps (same method as columns).
Includes header/footer classification via gap-size heuristics.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-28 01:14:31 +01:00
Benjamin Admin
c7ae44ff17 feat(rag): add 42 new regulations to RAG overview + update collection totals
Some checks failed
CI / go-lint (push) Has been skipped
CI / python-lint (push) Has been skipped
CI / nodejs-lint (push) Has been skipped
CI / test-go-school (push) Successful in 33s
CI / test-go-edu-search (push) Successful in 25s
CI / test-python-klausur (push) Failing after 1m46s
CI / test-python-agent-core (push) Successful in 15s
CI / test-nodejs-website (push) Successful in 23s
New regulations across bp_compliance_ce (11), bp_compliance_gesetze (31),
and bp_compliance_datenschutz (1). Collection totals updated:
gesetze 58304, ce 18183, datenschutz 2448, total 103912.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-28 01:04:27 +01:00
Benjamin Admin
ce0815007e feat(ocr-pipeline): replace clustering column detection with whitespace-gap analysis
Column detection now uses vertical projection profiles to find whitespace
gaps between columns, then validates gaps against word bounding boxes to
prevent splitting through words. Old clustering algorithm extracted as
fallback (_detect_columns_by_clustering) for pages with < 2 detected gaps.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-28 00:36:28 +01:00
Benjamin Admin
b03cb0a1e6 Fix Landkarte tab crash: variable name shadowed isInRag function
Local variables named 'isInRag' shadowed the outer function, causing
"isInRag is not a function" error. Renamed to regInRag/codeInRag.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-28 00:01:01 +01:00
Benjamin Admin
5a45cbf605 Update RAG page: Chunks/Status columns use hardcoded data, Key Intersections show RAG status
Some checks failed
CI / go-lint (push) Has been skipped
CI / python-lint (push) Has been skipped
CI / nodejs-lint (push) Has been skipped
CI / test-go-school (push) Successful in 24s
CI / test-go-edu-search (push) Successful in 24s
CI / test-python-klausur (push) Failing after 1m36s
CI / test-python-agent-core (push) Successful in 16s
CI / test-nodejs-website (push) Successful in 15s
- Chunks column now uses getKnownChunks() instead of API-based getRegulationChunks()
- Status column uses isInRag() check (green/red) instead of ratio-based calculation
- Key Intersections chips show green/red with checkmark/cross based on RAG status

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-27 23:53:21 +01:00
Benjamin Admin
164b35c06a fix(ocr-pipeline): tighten page_ref constraints based on live testing
- Reduce left-side threshold from 35% to 20% of content width
- Strong language signal (eng/deu > 0.3) now prevents page_ref assignment
- Increase column_ignore word threshold from 3 to 8 for edge columns
- Apply language guard to Level 1 and Level 2 classification

Fixes: column with deu=0.921 was misclassified as page_ref because
reference score check ran before language analysis.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-27 23:33:11 +01:00
Benjamin Admin
2297f66edb feat(rag): Add RAG status indicators and 4 new EU regulations
Some checks failed
CI / go-lint (push) Has been skipped
CI / python-lint (push) Has been skipped
CI / nodejs-lint (push) Has been skipped
CI / test-go-school (push) Successful in 26s
CI / test-go-edu-search (push) Successful in 27s
CI / test-python-klausur (push) Failing after 1m39s
CI / test-python-agent-core (push) Successful in 17s
CI / test-nodejs-website (push) Successful in 23s
- Add REGULATIONS_IN_RAG Set tracking all 42 regulations currently in Qdrant
- Add 4 new regulation entries: E-Commerce-RL, Verbraucherrechte-RL,
  Digitale-Inhalte-RL, DMA (all ingested Feb 2026)
- Add RAG column to regulations table with green check/red x indicators
- Update Landkarte tab: green/x on industry cards, thematic clusters,
  and regulation matrix
- Replace old "Integrated Regulations" section with full RAG coverage overview
- Update hardcoded chunk counts (Templates: 7689, NiBiS: 7996)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-27 23:23:52 +01:00
Benjamin Admin
db8327f039 fix(ocr-pipeline): tune column detection based on GT comparison
Address 5 weaknesses found via ground-truth comparison on session df3548d1:
- Add column_ignore for edge columns with < 3 words (margin detection)
- Absorb tiny clusters (< 5% width) into neighbors post-merge
- Restrict page_ref to left 35% of content area across all 3 levels
- Loosen marker thresholds (width < 6%, words <= 15) and add strong
  marker score for very narrow non-edge columns (< 4%)
- Add EN/DE position tiebreaker when language signals are both weak

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-27 23:16:31 +01:00
Benjamin Admin
587b066a40 feat(ocr-pipeline): ground-truth comparison tool for column detection
Side-by-side view: auto result (readonly) vs GT editor where teacher
draws correct columns. Diff table shows Auto vs GT with IoU matching.
GT data persisted per session for algorithm tuning.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-27 22:48:37 +01:00
Benjamin Admin
03fa186fec fix(ocr-pipeline): increase merge distance to 6% for better column merging
Sub-alignments within a column (indented words, etc.) were 60-90px apart
and not getting merged at 3%. On a typical 5-col page (~1500px), 6% = ~90px
merges sub-alignments while keeping real column boundaries (~300px) separate.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-27 20:19:09 +01:00
Benjamin Admin
1040729874 fix(ocr-pipeline): avoid backslash in f-string for Python 3.11 compat
Use format() instead of nested f-strings with escaped quotes.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-27 20:06:20 +01:00
Benjamin Admin
4f37afa222 feat(ocr-pipeline): verticality filter for column detection
Clusters now track Y-positions of their words and filter by vertical
coverage (>=30% primary, >=15%+5words secondary) to reject noise from
indentations or page numbers. Merge distance widened to 3% content width.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-27 19:57:13 +01:00
Benjamin Admin
bb879a03a8 feat(ocr-pipeline): add column_ignore type for margins/empty areas
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-27 08:51:56 +01:00
Benjamin Admin
f535d3c967 fix(ocr-pipeline): manual editor layout + no re-detection on cached result
- ManualColumnEditor now uses grid-cols-2 layout (image left, controls right)
  matching the normal view size so the image doesn't zoom in
- StepColumnDetection only runs auto-detection when no cached result exists;
  revisiting step 3 loads cached columns without re-running detection

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-27 08:45:49 +01:00
Benjamin Admin
7a3570fe46 feat(ocr-pipeline): manual column editor for Step 3
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-27 08:27:54 +01:00
Benjamin Admin
1393a994f9 Flexible inhaltsbasierte Spaltenerkennung (2-Phasen)
Ersetzt hardcodierte Positionsregeln durch ein zweistufiges System:
Phase A erkennt Spaltengeometrie (Clustering), Phase B klassifiziert
Typen per Inhalt (Sprache/Rolle) mit 3-stufiger Fallback-Kette.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-26 23:33:35 +01:00
Benjamin Admin
cf27a95308 feat(ocr-pipeline): word-based 5-column detection for vocabulary pages
Replace projection-profile layout analysis with Tesseract word bounding
box clustering to detect 5-column vocabulary layouts (page_ref, EN, DE,
markers, examples). Falls back to projection profiles when < 3 clusters.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-26 23:08:14 +01:00
Benjamin Admin
aa06ae0f61 feat: Persistente Sessions (PostgreSQL) + Spaltenerkennung (Step 3)
Sessions werden jetzt in PostgreSQL gespeichert statt in-memory.
Neue Session-Liste mit Name, Datum, Schritt. Sessions ueberleben
Browser-Refresh und Container-Neustart. Step 3 nutzt analyze_layout()
fuer automatische Spaltenerkennung mit farbigem Overlay.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-26 22:16:37 +01:00
Benjamin Admin
09b820efbe refactor(dewarp): replace displacement map with affine shear correction
The old displacement-map approach shifted entire rows by a parabolic
profile, creating a circle/barrel distortion. The actual problem is
a linear vertical shear: after deskew aligns horizontal lines, the
vertical column edges are still tilted by ~0.5°.

New approach:
- Detect shear angle from strongest vertical edge slope (not curvature)
- Apply cv2.warpAffine shear to straighten vertical features
- Manual slider: -2.0° to +2.0° in 0.05° steps
- Slider initializes to auto-detected shear angle
- Ground truth question: "Spalten vertikal ausgerichtet?"

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-26 18:23:04 +01:00
Benjamin Admin
ff2bb79a91 fix(dewarp): change manual slider to percentage (0-200%) instead of raw multiplier
The old -3.0 to +3.0 scale multiplied the full displacement map (up to ~79px)
directly, causing extreme distortion at values >1. New slider:
- 0% = no correction
- 100% = auto-detected correction (default)
- 200% = double correction
- Step size: 5%

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-26 18:10:34 +01:00
Benjamin Admin
fb496c5e34 perf(klausur-service): split Dockerfile into base + app layer
Tesseract OCR + 70 Debian packages + pip dependencies are now in a
separate base image (klausur-base:latest) that is built once and reused.
A --no-cache build now only rebuilds the code layer (~seconds) instead
of re-downloading 33 MB of system packages (~9 minutes).

Rebuild base when requirements.txt or system deps change:
  docker build -f klausur-service/Dockerfile.base -t klausur-base:latest klausur-service/

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-26 17:43:24 +01:00
Benjamin Admin
9df745574b fix(ocr-pipeline): dewarp visibility, grid on both sides, session persistence
- Fix dewarp method selection: prefer methods with >5px curvature over
  higher confidence (vertical_edge 79px was being ignored for text_baseline 2px)
- Add grid overlay on left image in Dewarp step for side-by-side comparison
- Add GET /sessions/{id} endpoint to reload session data
- StepDeskew accepts sessionId prop to restore state when navigating back
- SessionInfo type extended with optional deskew_result and dewarp_result

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-26 17:29:53 +01:00
Benjamin Admin
44e8c573af fix: Deskew Ground Truth Frage auf Rotation beschraenken
"Korrekt ausgerichtet?" → "Rotation korrekt?" mit Hinweis,
dass Woelbung/Verzerrung im naechsten Schritt korrigiert wird.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-26 17:16:24 +01:00
Benjamin Admin
589d2f811a feat: Dewarp-Korrektur als Schritt 2 in OCR Pipeline (7 Schritte)
Implementiert Buchwoelbungs-Entzerrung mit zwei Methoden:
- Methode A: Vertikale-Kanten-Analyse (Sobel + Polynom 2. Grades)
- Methode B: Textzeilen-Baseline (Tesseract + Baseline-Kruemmung)
Beste Methode wird automatisch gewaehlt, manueller Slider (-3 bis +3).

Backend: 3 neue Endpoints (auto/manual dewarp, ground truth)
Frontend: StepDewarp + DewarpControls, Pipeline von 6 auf 7 Schritte

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-26 16:46:41 +01:00
Benjamin Admin
d552fd8b6b feat: OCR Pipeline mit 6-Schritt-Wizard fuer Seitenrekonstruktion
All checks were successful
CI / go-lint (push) Has been skipped
CI / python-lint (push) Has been skipped
CI / nodejs-lint (push) Has been skipped
CI / test-go-school (push) Successful in 38s
CI / test-go-edu-search (push) Successful in 29s
CI / test-python-klausur (push) Successful in 1m46s
CI / test-python-agent-core (push) Successful in 17s
CI / test-nodejs-website (push) Successful in 22s
Neue Route /ai/ocr-pipeline mit schrittweiser Begradigung (Deskew),
Raster-Overlay und Ground Truth. Schritte 2-6 als Platzhalter.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-26 15:38:08 +01:00
Benjamin Admin
e7b6654b85 docs: update CLAUDE.md for direct MacBook development workflow
All checks were successful
CI / go-lint (push) Has been skipped
CI / python-lint (push) Has been skipped
CI / nodejs-lint (push) Has been skipped
CI / test-go-school (push) Successful in 27s
CI / test-go-edu-search (push) Successful in 27s
CI / test-python-klausur (push) Successful in 1m43s
CI / test-python-agent-core (push) Successful in 17s
CI / test-nodejs-website (push) Successful in 25s
Remove rsync-based workflow, document git push + Mac Mini pull workflow.
2026-02-25 23:09:42 +01:00
46 changed files with 16837 additions and 134 deletions

View File

@@ -6,19 +6,26 @@
| Geraet | Rolle | Aufgaben |
|--------|-------|----------|
| **MacBook** | Client | Claude Terminal, Browser (Frontend-Tests) |
| **Mac Mini** | Server | Docker, alle Services, Code-Ausfuehrung, Tests, Git |
| **MacBook** | Entwicklung | Claude Terminal, Code-Entwicklung, Browser (Frontend-Tests) |
| **Mac Mini** | Server | Docker, alle Services, Tests, Builds, Deployment |
**WICHTIG:** Die Entwicklung findet vollstaendig auf dem **Mac Mini** statt!
**WICHTIG:** Code wird direkt auf dem MacBook in diesem Repo bearbeitet. Docker und Services laufen auf dem Mac Mini.
### SSH-Verbindung
### Entwicklungsworkflow
```bash
ssh macmini
# Projektverzeichnis:
cd /Users/benjaminadmin/Projekte/breakpilot-lehrer
# 1. Code auf MacBook bearbeiten (dieses Verzeichnis)
# 2. Committen und pushen:
git push origin main && git push gitea main
# Einzelbefehle (BEVORZUGT):
# 3. Auf Mac Mini pullen und Container neu bauen:
ssh macmini "cd /Users/benjaminadmin/Projekte/breakpilot-lehrer && git pull --no-rebase origin main"
ssh macmini "cd /Users/benjaminadmin/Projekte/breakpilot-lehrer && /usr/local/bin/docker compose build --no-cache <service> && /usr/local/bin/docker compose up -d <service>"
```
### SSH-Verbindung (fuer Docker/Tests)
```bash
ssh macmini "cd /Users/benjaminadmin/Projekte/breakpilot-lehrer && <cmd>"
```

View File

@@ -0,0 +1,322 @@
'use client'
import { useCallback, useEffect, useState } from 'react'
import { PagePurpose } from '@/components/common/PagePurpose'
import { PipelineStepper } from '@/components/ocr-pipeline/PipelineStepper'
import { StepDeskew } from '@/components/ocr-pipeline/StepDeskew'
import { StepDewarp } from '@/components/ocr-pipeline/StepDewarp'
import { StepColumnDetection } from '@/components/ocr-pipeline/StepColumnDetection'
import { StepRowDetection } from '@/components/ocr-pipeline/StepRowDetection'
import { StepWordRecognition } from '@/components/ocr-pipeline/StepWordRecognition'
import { StepLlmReview } from '@/components/ocr-pipeline/StepLlmReview'
import { StepReconstruction } from '@/components/ocr-pipeline/StepReconstruction'
import { StepGroundTruth } from '@/components/ocr-pipeline/StepGroundTruth'
import { PIPELINE_STEPS, type PipelineStep, type SessionListItem } from './types'
const KLAUSUR_API = '/klausur-api'
export default function OcrPipelinePage() {
const [currentStep, setCurrentStep] = useState(0)
const [sessionId, setSessionId] = useState<string | null>(null)
const [sessionName, setSessionName] = useState<string>('')
const [sessions, setSessions] = useState<SessionListItem[]>([])
const [loadingSessions, setLoadingSessions] = useState(true)
const [editingName, setEditingName] = useState<string | null>(null)
const [editNameValue, setEditNameValue] = useState('')
const [steps, setSteps] = useState<PipelineStep[]>(
PIPELINE_STEPS.map((s, i) => ({
...s,
status: i === 0 ? 'active' : 'pending',
})),
)
// Load session list on mount
useEffect(() => {
loadSessions()
}, [])
const loadSessions = async () => {
setLoadingSessions(true)
try {
const res = await fetch(`${KLAUSUR_API}/api/v1/ocr-pipeline/sessions`)
if (res.ok) {
const data = await res.json()
setSessions(data.sessions || [])
}
} catch (e) {
console.error('Failed to load sessions:', e)
} finally {
setLoadingSessions(false)
}
}
const openSession = useCallback(async (sid: string) => {
try {
const res = await fetch(`${KLAUSUR_API}/api/v1/ocr-pipeline/sessions/${sid}`)
if (!res.ok) return
const data = await res.json()
setSessionId(sid)
setSessionName(data.name || data.filename || '')
// Determine which step to jump to based on current_step
const dbStep = data.current_step || 1
// Steps: 1=deskew, 2=dewarp, 3=columns, ...
// UI steps are 0-indexed: 0=deskew, 1=dewarp, 2=columns, ...
const uiStep = Math.max(0, dbStep - 1)
setSteps(
PIPELINE_STEPS.map((s, i) => ({
...s,
status: i < uiStep ? 'completed' : i === uiStep ? 'active' : 'pending',
})),
)
setCurrentStep(uiStep)
} catch (e) {
console.error('Failed to open session:', e)
}
}, [])
const deleteSession = useCallback(async (sid: string) => {
try {
await fetch(`${KLAUSUR_API}/api/v1/ocr-pipeline/sessions/${sid}`, { method: 'DELETE' })
setSessions((prev) => prev.filter((s) => s.id !== sid))
if (sessionId === sid) {
setSessionId(null)
setCurrentStep(0)
setSteps(PIPELINE_STEPS.map((s, i) => ({ ...s, status: i === 0 ? 'active' : 'pending' })))
}
} catch (e) {
console.error('Failed to delete session:', e)
}
}, [sessionId])
const renameSession = useCallback(async (sid: string, newName: string) => {
try {
await fetch(`${KLAUSUR_API}/api/v1/ocr-pipeline/sessions/${sid}`, {
method: 'PUT',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ name: newName }),
})
setSessions((prev) => prev.map((s) => (s.id === sid ? { ...s, name: newName } : s)))
if (sessionId === sid) setSessionName(newName)
} catch (e) {
console.error('Failed to rename session:', e)
}
setEditingName(null)
}, [sessionId])
const handleStepClick = (index: number) => {
if (index <= currentStep || steps[index].status === 'completed') {
setCurrentStep(index)
}
}
const goToStep = (step: number) => {
setCurrentStep(step)
setSteps((prev) =>
prev.map((s, i) => ({
...s,
status: i < step ? 'completed' : i === step ? 'active' : 'pending',
})),
)
}
const handleNext = () => {
if (currentStep < steps.length - 1) {
setSteps((prev) =>
prev.map((s, i) => {
if (i === currentStep) return { ...s, status: 'completed' }
if (i === currentStep + 1) return { ...s, status: 'active' }
return s
}),
)
setCurrentStep((prev) => prev + 1)
}
}
const handleDeskewComplete = (sid: string) => {
setSessionId(sid)
// Reload session list to show the new session
loadSessions()
handleNext()
}
const handleNewSession = () => {
setSessionId(null)
setSessionName('')
setCurrentStep(0)
setSteps(PIPELINE_STEPS.map((s, i) => ({ ...s, status: i === 0 ? 'active' : 'pending' })))
}
const stepNames: Record<number, string> = {
1: 'Begradigung',
2: 'Entzerrung',
3: 'Spalten',
4: 'Zeilen',
5: 'Woerter',
6: 'LLM-Korrektur',
7: 'Rekonstruktion',
8: 'Validierung',
}
const reprocessFromStep = useCallback(async (uiStep: number) => {
if (!sessionId) return
const dbStep = uiStep + 1 // UI is 0-indexed, DB is 1-indexed
if (!confirm(`Ab Schritt ${dbStep} (${stepNames[dbStep] || '?'}) neu verarbeiten? Nachfolgende Daten werden geloescht.`)) return
try {
const res = await fetch(`${KLAUSUR_API}/api/v1/ocr-pipeline/sessions/${sessionId}/reprocess`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ from_step: dbStep }),
})
if (!res.ok) {
const data = await res.json().catch(() => ({}))
console.error('Reprocess failed:', data.detail || res.status)
return
}
// Reset UI steps
goToStep(uiStep)
} catch (e) {
console.error('Reprocess error:', e)
}
// eslint-disable-next-line react-hooks/exhaustive-deps
}, [sessionId, goToStep])
const renderStep = () => {
switch (currentStep) {
case 0:
return <StepDeskew sessionId={sessionId} onNext={handleDeskewComplete} />
case 1:
return <StepDewarp sessionId={sessionId} onNext={handleNext} />
case 2:
return <StepColumnDetection sessionId={sessionId} onNext={handleNext} />
case 3:
return <StepRowDetection sessionId={sessionId} onNext={handleNext} />
case 4:
return <StepWordRecognition sessionId={sessionId} onNext={handleNext} goToStep={goToStep} />
case 5:
return <StepLlmReview sessionId={sessionId} onNext={handleNext} />
case 6:
return <StepReconstruction sessionId={sessionId} onNext={handleNext} />
case 7:
return <StepGroundTruth />
default:
return null
}
}
return (
<div className="space-y-6">
<PagePurpose
title="OCR Pipeline"
purpose="Schrittweise Seitenrekonstruktion: Scan begradigen, Spalten erkennen, Woerter lokalisieren und die Seite Wort fuer Wort nachbauen. Ziel: 10 Vokabelseiten fehlerfrei rekonstruieren."
audience={['Entwickler', 'Data Scientists']}
architecture={{
services: ['klausur-service (FastAPI)', 'OpenCV', 'Tesseract'],
databases: ['PostgreSQL Sessions'],
}}
relatedPages={[
{ name: 'OCR Vergleich', href: '/ai/ocr-compare', description: 'Methoden-Vergleich' },
{ name: 'OCR-Labeling', href: '/ai/ocr-labeling', description: 'Trainingsdaten' },
]}
defaultCollapsed
/>
{/* Session List */}
<div className="bg-white dark:bg-gray-800 rounded-xl border border-gray-200 dark:border-gray-700 p-4">
<div className="flex items-center justify-between mb-3">
<h3 className="text-sm font-medium text-gray-700 dark:text-gray-300">
Sessions
</h3>
<button
onClick={handleNewSession}
className="text-xs px-3 py-1.5 bg-teal-600 text-white rounded-lg hover:bg-teal-700 transition-colors"
>
+ Neue Session
</button>
</div>
{loadingSessions ? (
<div className="text-sm text-gray-400 py-2">Lade Sessions...</div>
) : sessions.length === 0 ? (
<div className="text-sm text-gray-400 py-2">Noch keine Sessions vorhanden.</div>
) : (
<div className="space-y-1 max-h-48 overflow-y-auto">
{sessions.map((s) => (
<div
key={s.id}
className={`flex items-center gap-2 px-3 py-2 rounded-lg text-sm transition-colors cursor-pointer ${
sessionId === s.id
? 'bg-teal-50 dark:bg-teal-900/30 border border-teal-200 dark:border-teal-700'
: 'hover:bg-gray-50 dark:hover:bg-gray-700/50'
}`}
>
<div className="flex-1 min-w-0" onClick={() => openSession(s.id)}>
{editingName === s.id ? (
<input
autoFocus
value={editNameValue}
onChange={(e) => setEditNameValue(e.target.value)}
onBlur={() => renameSession(s.id, editNameValue)}
onKeyDown={(e) => {
if (e.key === 'Enter') renameSession(s.id, editNameValue)
if (e.key === 'Escape') setEditingName(null)
}}
onClick={(e) => e.stopPropagation()}
className="w-full px-1 py-0.5 text-sm border rounded dark:bg-gray-700 dark:border-gray-600"
/>
) : (
<div className="truncate font-medium text-gray-700 dark:text-gray-300">
{s.name || s.filename}
</div>
)}
<div className="text-xs text-gray-400 flex gap-2">
<span>{new Date(s.created_at).toLocaleDateString('de-DE', { day: '2-digit', month: '2-digit', year: '2-digit', hour: '2-digit', minute: '2-digit' })}</span>
<span>Schritt {s.current_step}: {stepNames[s.current_step] || '?'}</span>
</div>
</div>
<button
onClick={(e) => {
e.stopPropagation()
setEditNameValue(s.name || s.filename)
setEditingName(s.id)
}}
className="p-1 text-gray-400 hover:text-gray-600 dark:hover:text-gray-300"
title="Umbenennen"
>
<svg className="w-3.5 h-3.5" fill="none" viewBox="0 0 24 24" stroke="currentColor" strokeWidth={2}>
<path strokeLinecap="round" strokeLinejoin="round" d="M15.232 5.232l3.536 3.536m-2.036-5.036a2.5 2.5 0 113.536 3.536L6.5 21.036H3v-3.572L16.732 3.732z" />
</svg>
</button>
<button
onClick={(e) => {
e.stopPropagation()
if (confirm('Session loeschen?')) deleteSession(s.id)
}}
className="p-1 text-gray-400 hover:text-red-500"
title="Loeschen"
>
<svg className="w-3.5 h-3.5" fill="none" viewBox="0 0 24 24" stroke="currentColor" strokeWidth={2}>
<path strokeLinecap="round" strokeLinejoin="round" d="M19 7l-.867 12.142A2 2 0 0116.138 21H7.862a2 2 0 01-1.995-1.858L5 7m5 4v6m4-6v6m1-10V4a1 1 0 00-1-1h-4a1 1 0 00-1 1v3M4 7h16" />
</svg>
</button>
</div>
))}
</div>
)}
</div>
{/* Active session name */}
{sessionId && sessionName && (
<div className="text-sm text-gray-500 dark:text-gray-400">
Aktive Session: <span className="font-medium text-gray-700 dark:text-gray-300">{sessionName}</span>
</div>
)}
<PipelineStepper steps={steps} currentStep={currentStep} onStepClick={handleStepClick} onReprocess={sessionId ? reprocessFromStep : undefined} />
<div className="min-h-[400px]">{renderStep()}</div>
</div>
)
}

View File

@@ -0,0 +1,222 @@
export type PipelineStepStatus = 'pending' | 'active' | 'completed' | 'failed'
export interface PipelineStep {
id: string
name: string
icon: string
status: PipelineStepStatus
}
export interface SessionListItem {
id: string
name: string
filename: string
status: string
current_step: number
created_at: string
updated_at?: string
}
export interface SessionInfo {
session_id: string
filename: string
name?: string
image_width: number
image_height: number
original_image_url: string
current_step?: number
deskew_result?: DeskewResult
dewarp_result?: DewarpResult
column_result?: ColumnResult
row_result?: RowResult
word_result?: GridResult
}
export interface DeskewResult {
session_id: string
angle_hough: number
angle_word_alignment: number
angle_applied: number
method_used: 'hough' | 'word_alignment' | 'manual'
confidence: number
duration_seconds: number
deskewed_image_url: string
binarized_image_url: string
}
export interface DeskewGroundTruth {
is_correct: boolean
corrected_angle?: number
notes?: string
}
export interface DewarpResult {
session_id: string
method_used: 'vertical_edge' | 'manual' | 'none'
shear_degrees: number
confidence: number
duration_seconds: number
dewarped_image_url: string
}
export interface DewarpGroundTruth {
is_correct: boolean
corrected_shear?: number
notes?: string
}
export interface PageRegion {
type: 'column_en' | 'column_de' | 'column_example' | 'page_ref'
| 'column_marker' | 'column_text' | 'column_ignore' | 'header' | 'footer'
x: number
y: number
width: number
height: number
classification_confidence?: number
classification_method?: string
}
export interface ColumnResult {
columns: PageRegion[]
duration_seconds: number
}
export interface ColumnGroundTruth {
is_correct: boolean
corrected_columns?: PageRegion[]
notes?: string
}
export interface ManualColumnDivider {
xPercent: number // Position in % of image width (0-100)
}
export type ColumnTypeKey = PageRegion['type']
export interface RowResult {
rows: RowItem[]
summary: Record<string, number>
total_rows: number
duration_seconds: number
}
export interface RowItem {
index: number
x: number
y: number
width: number
height: number
word_count: number
row_type: 'content' | 'header' | 'footer'
gap_before: number
}
export interface RowGroundTruth {
is_correct: boolean
corrected_rows?: RowItem[]
notes?: string
}
export interface WordBbox {
x: number
y: number
w: number
h: number
}
export interface GridCell {
cell_id: string // "R03_C1"
row_index: number
col_index: number
col_type: string
text: string
confidence: number
bbox_px: WordBbox
bbox_pct: WordBbox
ocr_engine?: string
status?: 'pending' | 'confirmed' | 'edited' | 'skipped'
}
export interface ColumnMeta {
index: number
type: string
x: number
width: number
}
export interface GridResult {
cells: GridCell[]
grid_shape: { rows: number; cols: number; total_cells: number }
columns_used: ColumnMeta[]
layout: 'vocab' | 'generic'
image_width: number
image_height: number
duration_seconds: number
ocr_engine?: string
vocab_entries?: WordEntry[] // Only when layout='vocab'
entries?: WordEntry[] // Backwards compat alias for vocab_entries
entry_count?: number
summary: {
total_cells: number
non_empty_cells: number
low_confidence: number
// Only when layout='vocab':
total_entries?: number
with_english?: number
with_german?: number
}
llm_review?: {
changes: { row_index: number; field: string; old: string; new: string }[]
model_used: string
duration_ms: number
entries_corrected: number
applied_count?: number
applied_at?: string
}
}
export interface WordEntry {
row_index: number
english: string
german: string
example: string
confidence: number
bbox: WordBbox
bbox_en: WordBbox | null
bbox_de: WordBbox | null
bbox_ex: WordBbox | null
status?: 'pending' | 'confirmed' | 'edited' | 'skipped'
}
/** @deprecated Use GridResult instead */
export interface WordResult {
entries: WordEntry[]
entry_count: number
image_width: number
image_height: number
duration_seconds: number
ocr_engine?: string
summary: {
total_entries: number
with_english: number
with_german: number
low_confidence: number
}
}
export interface WordGroundTruth {
is_correct: boolean
corrected_entries?: WordEntry[]
notes?: string
}
export const PIPELINE_STEPS: PipelineStep[] = [
{ id: 'deskew', name: 'Begradigung', icon: '📐', status: 'pending' },
{ id: 'dewarp', name: 'Entzerrung', icon: '🔧', status: 'pending' },
{ id: 'columns', name: 'Spalten', icon: '📊', status: 'pending' },
{ id: 'rows', name: 'Zeilen', icon: '📏', status: 'pending' },
{ id: 'words', name: 'Woerter', icon: '🔤', status: 'pending' },
{ id: 'llm-review', name: 'LLM-Korrektur', icon: '🤖', status: 'pending' },
{ id: 'reconstruction', name: 'Rekonstruktion', icon: '🏗️', status: 'pending' },
{ id: 'ground-truth', name: 'Validierung', icon: '✅', status: 'pending' },
]

View File

@@ -0,0 +1,671 @@
'use client'
import React, { useState, useEffect, useCallback, useRef } from 'react'
import { RAG_PDF_MAPPING } from './rag-pdf-mapping'
import { REGULATIONS_IN_RAG, REGULATION_INFO } from '../rag-constants'
interface ChunkBrowserQAProps {
apiProxy: string
}
type RegGroupKey = 'eu_regulation' | 'eu_directive' | 'de_law' | 'at_law' | 'ch_law' | 'national_law' | 'bsi_standard' | 'eu_guideline' | 'international_standard' | 'other'
const GROUP_LABELS: Record<RegGroupKey, string> = {
eu_regulation: 'EU Verordnungen',
eu_directive: 'EU Richtlinien',
de_law: 'DE Gesetze',
at_law: 'AT Gesetze',
ch_law: 'CH Gesetze',
national_law: 'Nationale Gesetze (EU)',
bsi_standard: 'BSI Standards',
eu_guideline: 'EDPB / Guidelines',
international_standard: 'Internationale Standards',
other: 'Sonstige',
}
const GROUP_ORDER: RegGroupKey[] = [
'eu_regulation', 'eu_directive', 'de_law', 'at_law', 'ch_law',
'national_law', 'bsi_standard', 'eu_guideline', 'international_standard', 'other',
]
const COLLECTIONS = [
'bp_compliance_gesetze',
'bp_compliance_ce',
'bp_compliance_datenschutz',
]
export function ChunkBrowserQA({ apiProxy }: ChunkBrowserQAProps) {
// Filter-Sidebar
const [selectedRegulation, setSelectedRegulation] = useState<string | null>(null)
const [regulationCounts, setRegulationCounts] = useState<Record<string, number>>({})
const [filterSearch, setFilterSearch] = useState('')
const [countsLoading, setCountsLoading] = useState(false)
// Dokument-Chunks (sequenziell)
const [docChunks, setDocChunks] = useState<Record<string, unknown>[]>([])
const [docChunkIndex, setDocChunkIndex] = useState(0)
const [docTotalChunks, setDocTotalChunks] = useState(0)
const [docLoading, setDocLoading] = useState(false)
const docChunksRef = useRef(docChunks)
docChunksRef.current = docChunks
// Split-View
const [splitViewActive, setSplitViewActive] = useState(true)
const [chunksPerPage, setChunksPerPage] = useState(6)
const [fullscreen, setFullscreen] = useState(false)
// Collection — default to bp_compliance_ce where we have PDFs downloaded
const [collection, setCollection] = useState('bp_compliance_ce')
// PDF existence check
const [pdfExists, setPdfExists] = useState<boolean | null>(null)
// Sidebar collapsed groups
const [collapsedGroups, setCollapsedGroups] = useState<Set<string>>(new Set())
// Build grouped regulations for sidebar
const regulationsInCollection = Object.entries(REGULATIONS_IN_RAG)
.filter(([, info]) => info.collection === collection)
.map(([code]) => code)
const groupedRegulations = React.useMemo(() => {
const groups: Record<RegGroupKey, { code: string; name: string; type: string }[]> = {
eu_regulation: [], eu_directive: [], de_law: [], at_law: [], ch_law: [],
national_law: [], bsi_standard: [], eu_guideline: [], international_standard: [], other: [],
}
for (const code of regulationsInCollection) {
const reg = REGULATION_INFO.find(r => r.code === code)
const type = (reg?.type || 'other') as RegGroupKey
const groupKey = type in groups ? type : 'other'
groups[groupKey].push({
code,
name: reg?.name || code,
type: reg?.type || 'unknown',
})
}
return groups
}, [regulationsInCollection.join(',')])
// Load regulation counts for current collection
const loadRegulationCounts = useCallback(async (col: string) => {
const entries = Object.entries(REGULATIONS_IN_RAG)
.filter(([, info]) => info.collection === col && info.qdrant_id)
if (entries.length === 0) return
// Build qdrant_id -> our_code mapping
const qdrantIdToCode: Record<string, string[]> = {}
for (const [code, info] of entries) {
if (!qdrantIdToCode[info.qdrant_id]) qdrantIdToCode[info.qdrant_id] = []
qdrantIdToCode[info.qdrant_id].push(code)
}
const uniqueQdrantIds = Object.keys(qdrantIdToCode)
setCountsLoading(true)
try {
const params = new URLSearchParams({
action: 'regulation-counts-batch',
collection: col,
qdrant_ids: uniqueQdrantIds.join(','),
})
const res = await fetch(`${apiProxy}?${params}`)
if (res.ok) {
const data = await res.json()
// Map qdrant_id counts back to our codes
const mapped: Record<string, number> = {}
for (const [qid, count] of Object.entries(data.counts as Record<string, number>)) {
const codes = qdrantIdToCode[qid] || []
for (const code of codes) {
mapped[code] = count
}
}
setRegulationCounts(prev => ({ ...prev, ...mapped }))
}
} catch (error) {
console.error('Failed to load regulation counts:', error)
} finally {
setCountsLoading(false)
}
}, [apiProxy])
// Load all chunks for a regulation (paginated scroll)
const loadDocumentChunks = useCallback(async (regulationCode: string) => {
const ragInfo = REGULATIONS_IN_RAG[regulationCode]
if (!ragInfo || !ragInfo.qdrant_id) return
setDocLoading(true)
setDocChunks([])
setDocChunkIndex(0)
setDocTotalChunks(0)
const allChunks: Record<string, unknown>[] = []
let offset: string | null = null
try {
let safety = 0
do {
const params = new URLSearchParams({
action: 'scroll',
collection: ragInfo.collection,
limit: '100',
filter_key: 'regulation_id',
filter_value: ragInfo.qdrant_id,
})
if (offset) params.append('offset', offset)
const res = await fetch(`${apiProxy}?${params}`)
if (!res.ok) break
const data = await res.json()
const chunks = data.chunks || []
allChunks.push(...chunks)
offset = data.next_offset || null
safety++
} while (offset && safety < 200)
// Sort by chunk_index
allChunks.sort((a, b) => {
const ai = Number(a.chunk_index ?? a.chunk_id ?? 0)
const bi = Number(b.chunk_index ?? b.chunk_id ?? 0)
return ai - bi
})
setDocChunks(allChunks)
setDocTotalChunks(allChunks.length)
setDocChunkIndex(0)
} catch (error) {
console.error('Failed to load document chunks:', error)
} finally {
setDocLoading(false)
}
}, [apiProxy])
// Initial load
useEffect(() => {
loadRegulationCounts(collection)
}, [collection, loadRegulationCounts])
// Current chunk
const currentChunk = docChunks[docChunkIndex] || null
const prevChunk = docChunkIndex > 0 ? docChunks[docChunkIndex - 1] : null
const nextChunk = docChunkIndex < docChunks.length - 1 ? docChunks[docChunkIndex + 1] : null
// PDF page estimation — use pages metadata if available
const estimatePdfPage = (chunk: Record<string, unknown> | null, chunkIdx: number): number => {
if (chunk) {
// Try pages array from payload (e.g. [7] or [7,8])
const pages = chunk.pages as number[] | undefined
if (Array.isArray(pages) && pages.length > 0) return pages[0]
// Try page field
const page = chunk.page as number | undefined
if (typeof page === 'number' && page > 0) return page
}
const mapping = selectedRegulation ? RAG_PDF_MAPPING[selectedRegulation] : null
const cpp = mapping?.chunksPerPage || chunksPerPage
return Math.floor(chunkIdx / cpp) + 1
}
const pdfPage = estimatePdfPage(currentChunk, docChunkIndex)
const pdfMapping = selectedRegulation ? RAG_PDF_MAPPING[selectedRegulation] : null
const pdfUrl = pdfMapping ? `/rag-originals/${pdfMapping.filename}#page=${pdfPage}` : null
// Check PDF existence when regulation changes
useEffect(() => {
if (!selectedRegulation) { setPdfExists(null); return }
const mapping = RAG_PDF_MAPPING[selectedRegulation]
if (!mapping) { setPdfExists(false); return }
const url = `/rag-originals/${mapping.filename}`
fetch(url, { method: 'HEAD' })
.then(res => setPdfExists(res.ok))
.catch(() => setPdfExists(false))
}, [selectedRegulation])
// Handlers
const handleSelectRegulation = (code: string) => {
setSelectedRegulation(code)
loadDocumentChunks(code)
}
const handleCollectionChange = (col: string) => {
setCollection(col)
setSelectedRegulation(null)
setDocChunks([])
setDocChunkIndex(0)
setDocTotalChunks(0)
setRegulationCounts({})
}
const handlePrev = () => {
if (docChunkIndex > 0) setDocChunkIndex(i => i - 1)
}
const handleNext = () => {
if (docChunkIndex < docChunks.length - 1) setDocChunkIndex(i => i + 1)
}
const handleKeyDown = useCallback((e: KeyboardEvent) => {
if (e.key === 'Escape' && fullscreen) {
e.preventDefault()
setFullscreen(false)
} else if (e.key === 'ArrowLeft' || e.key === 'ArrowUp') {
e.preventDefault()
setDocChunkIndex(i => Math.max(0, i - 1))
} else if (e.key === 'ArrowRight' || e.key === 'ArrowDown') {
e.preventDefault()
setDocChunkIndex(i => Math.min(docChunksRef.current.length - 1, i + 1))
}
}, [fullscreen])
useEffect(() => {
if (fullscreen || (selectedRegulation && docChunks.length > 0)) {
window.addEventListener('keydown', handleKeyDown)
return () => window.removeEventListener('keydown', handleKeyDown)
}
}, [selectedRegulation, docChunks.length, handleKeyDown, fullscreen])
const toggleGroup = (group: string) => {
setCollapsedGroups(prev => {
const next = new Set(prev)
if (next.has(group)) next.delete(group)
else next.add(group)
return next
})
}
// Get text content from a chunk
const getChunkText = (chunk: Record<string, unknown> | null): string => {
if (!chunk) return ''
return String(chunk.chunk_text || chunk.text || chunk.content || '')
}
// Extract structural metadata for prominent display
const getStructuralInfo = (chunk: Record<string, unknown> | null): { article?: string; section?: string; pages?: string } => {
if (!chunk) return {}
const result: { article?: string; section?: string; pages?: string } = {}
// Article / paragraph
const article = chunk.article || chunk.artikel || chunk.paragraph || chunk.section_title
if (article) result.article = String(article)
// Section
const section = chunk.section || chunk.chapter || chunk.abschnitt || chunk.kapitel
if (section) result.section = String(section)
// Pages
const pages = chunk.pages as number[] | undefined
if (Array.isArray(pages) && pages.length > 0) {
result.pages = pages.length === 1 ? `S. ${pages[0]}` : `S. ${pages[0]}-${pages[pages.length - 1]}`
} else if (chunk.page) {
result.pages = `S. ${chunk.page}`
}
return result
}
// Overlap extraction
const getOverlapPrev = (): string => {
if (!prevChunk) return ''
const text = getChunkText(prevChunk)
return text.length > 150 ? '...' + text.slice(-150) : text
}
const getOverlapNext = (): string => {
if (!nextChunk) return ''
const text = getChunkText(nextChunk)
return text.length > 150 ? text.slice(0, 150) + '...' : text
}
// Filter sidebar items
const filteredRegulations = React.useMemo(() => {
if (!filterSearch.trim()) return groupedRegulations
const term = filterSearch.toLowerCase()
const filtered: typeof groupedRegulations = {
eu_regulation: [], eu_directive: [], de_law: [], at_law: [], ch_law: [],
national_law: [], bsi_standard: [], eu_guideline: [], international_standard: [], other: [],
}
for (const [group, items] of Object.entries(groupedRegulations)) {
filtered[group as RegGroupKey] = items.filter(
r => r.code.toLowerCase().includes(term) || r.name.toLowerCase().includes(term)
)
}
return filtered
}, [groupedRegulations, filterSearch])
// Regulation name lookup
const getRegName = (code: string): string => {
const reg = REGULATION_INFO.find(r => r.code === code)
return reg?.name || code
}
// Important metadata keys to show prominently
const STRUCTURAL_KEYS = new Set([
'article', 'artikel', 'paragraph', 'section_title', 'section', 'chapter',
'abschnitt', 'kapitel', 'pages', 'page',
])
const HIDDEN_KEYS = new Set([
'text', 'content', 'chunk_text', 'id', 'embedding',
])
const structInfo = getStructuralInfo(currentChunk)
return (
<div
className={`flex flex-col ${fullscreen ? 'fixed inset-0 z-50 bg-slate-100 p-4' : ''}`}
style={fullscreen ? { height: '100vh' } : { height: 'calc(100vh - 220px)' }}
>
{/* Header bar — fixed height */}
<div className="flex-shrink-0 bg-white rounded-xl border border-slate-200 p-3 mb-3">
<div className="flex flex-wrap items-center gap-4">
<div>
<label className="block text-xs font-medium text-slate-500 mb-1">Collection</label>
<select
value={collection}
onChange={(e) => handleCollectionChange(e.target.value)}
className="px-3 py-1.5 border rounded-lg text-sm focus:ring-2 focus:ring-teal-500"
>
{COLLECTIONS.map(c => (
<option key={c} value={c}>{c}</option>
))}
</select>
</div>
{selectedRegulation && (
<>
<div className="flex items-center gap-2">
<span className="text-sm font-semibold text-slate-900">
{selectedRegulation} {getRegName(selectedRegulation)}
</span>
{structInfo.article && (
<span className="px-2 py-0.5 bg-blue-100 text-blue-800 text-xs font-medium rounded">
{structInfo.article}
</span>
)}
{structInfo.pages && (
<span className="px-2 py-0.5 bg-slate-100 text-slate-600 text-xs rounded">
{structInfo.pages}
</span>
)}
</div>
<div className="flex items-center gap-2 ml-auto">
<button
onClick={handlePrev}
disabled={docChunkIndex === 0}
className="px-3 py-1.5 text-sm font-medium border rounded-lg bg-white hover:bg-slate-50 disabled:opacity-30 disabled:cursor-not-allowed"
>
&#9664; Zurueck
</button>
<span className="text-sm font-mono text-slate-600 min-w-[80px] text-center">
{docChunkIndex + 1} / {docTotalChunks}
</span>
<button
onClick={handleNext}
disabled={docChunkIndex >= docChunks.length - 1}
className="px-3 py-1.5 text-sm font-medium border rounded-lg bg-white hover:bg-slate-50 disabled:opacity-30 disabled:cursor-not-allowed"
>
Weiter &#9654;
</button>
<input
type="number"
min={1}
max={docTotalChunks}
value={docChunkIndex + 1}
onChange={(e) => {
const v = parseInt(e.target.value, 10)
if (!isNaN(v) && v >= 1 && v <= docTotalChunks) setDocChunkIndex(v - 1)
}}
className="w-16 px-2 py-1 border rounded text-xs text-center"
title="Springe zu Chunk Nr."
/>
</div>
<div className="flex items-center gap-2">
<label className="text-xs text-slate-500">Chunks/Seite:</label>
<select
value={chunksPerPage}
onChange={(e) => setChunksPerPage(Number(e.target.value))}
className="px-2 py-1 border rounded text-xs"
>
{[3, 4, 5, 6, 8, 10, 12, 15, 20].map(n => (
<option key={n} value={n}>{n}</option>
))}
</select>
<button
onClick={() => setSplitViewActive(!splitViewActive)}
className={`px-3 py-1 text-xs rounded-lg border ${
splitViewActive ? 'bg-teal-50 border-teal-300 text-teal-700' : 'bg-slate-50 border-slate-300 text-slate-600'
}`}
>
{splitViewActive ? 'Split-View an' : 'Split-View aus'}
</button>
<button
onClick={() => setFullscreen(!fullscreen)}
className={`px-3 py-1 text-xs rounded-lg border ${
fullscreen ? 'bg-indigo-50 border-indigo-300 text-indigo-700' : 'bg-slate-50 border-slate-300 text-slate-600'
}`}
title={fullscreen ? 'Vollbild beenden (Esc)' : 'Vollbild'}
>
{fullscreen ? '&#10005; Vollbild beenden' : '&#9974; Vollbild'}
</button>
</div>
</>
)}
</div>
</div>
{/* Main content: Sidebar + Content — fills remaining height */}
<div className="flex gap-3 flex-1 min-h-0">
{/* Sidebar — scrollable */}
<div className="w-56 flex-shrink-0 bg-white rounded-xl border border-slate-200 flex flex-col min-h-0">
<div className="flex-shrink-0 p-3 border-b border-slate-100">
<input
type="text"
value={filterSearch}
onChange={(e) => setFilterSearch(e.target.value)}
placeholder="Suche..."
className="w-full px-2 py-1.5 border rounded-lg text-sm focus:ring-2 focus:ring-teal-500"
/>
{countsLoading && (
<div className="text-xs text-slate-400 mt-1 animate-pulse">Counts laden...</div>
)}
</div>
<div className="flex-1 overflow-y-auto min-h-0">
{GROUP_ORDER.map(group => {
const items = filteredRegulations[group]
if (items.length === 0) return null
const isCollapsed = collapsedGroups.has(group)
return (
<div key={group}>
<button
onClick={() => toggleGroup(group)}
className="w-full px-3 py-1.5 text-left text-xs font-semibold text-slate-500 bg-slate-50 hover:bg-slate-100 flex items-center justify-between sticky top-0 z-10"
>
<span>{GROUP_LABELS[group]}</span>
<span className="text-slate-400">{isCollapsed ? '+' : '-'}</span>
</button>
{!isCollapsed && items.map(reg => {
const count = regulationCounts[reg.code] ?? 0
const isSelected = selectedRegulation === reg.code
return (
<button
key={reg.code}
onClick={() => handleSelectRegulation(reg.code)}
className={`w-full px-3 py-1.5 text-left text-sm flex items-center justify-between hover:bg-teal-50 transition-colors ${
isSelected ? 'bg-teal-100 text-teal-900 font-medium' : 'text-slate-700'
}`}
>
<span className="truncate text-xs">{reg.name || reg.code}</span>
<span className={`text-xs tabular-nums flex-shrink-0 ml-1 ${count > 0 ? 'text-slate-500' : 'text-slate-300'}`}>
{count > 0 ? count.toLocaleString() : '—'}
</span>
</button>
)
})}
</div>
)
})}
</div>
</div>
{/* Content area — fills remaining width and height */}
{!selectedRegulation ? (
<div className="flex-1 flex items-center justify-center bg-white rounded-xl border border-slate-200">
<div className="text-center text-slate-400 space-y-2">
<div className="text-4xl">&#128269;</div>
<p className="text-sm">Dokument in der Sidebar auswaehlen, um QA zu starten.</p>
<p className="text-xs text-slate-300">Pfeiltasten: Chunk vor/zurueck</p>
</div>
</div>
) : docLoading ? (
<div className="flex-1 flex items-center justify-center bg-white rounded-xl border border-slate-200">
<div className="text-center text-slate-500 space-y-2">
<div className="animate-spin text-3xl">&#9881;</div>
<p className="text-sm">Chunks werden geladen...</p>
<p className="text-xs text-slate-400">
{selectedRegulation}: {REGULATIONS_IN_RAG[selectedRegulation]?.chunks.toLocaleString() || '?'} Chunks erwartet
</p>
</div>
</div>
) : (
<div className={`flex-1 grid gap-3 min-h-0 ${splitViewActive ? 'grid-cols-2' : 'grid-cols-1'}`}>
{/* Chunk-Text Panel — fixed height, internal scroll */}
<div className="bg-white rounded-xl border border-slate-200 flex flex-col min-h-0 overflow-hidden">
{/* Panel header */}
<div className="flex-shrink-0 px-4 py-2 bg-slate-50 border-b border-slate-100 flex items-center justify-between">
<span className="text-sm font-medium text-slate-700">Chunk-Text</span>
<div className="flex items-center gap-2">
{structInfo.article && (
<span className="px-2 py-0.5 bg-blue-50 text-blue-700 text-xs font-medium rounded border border-blue-200">
{structInfo.article}
</span>
)}
{structInfo.section && (
<span className="px-2 py-0.5 bg-purple-50 text-purple-700 text-xs rounded border border-purple-200">
{structInfo.section}
</span>
)}
<span className="text-xs text-slate-400 tabular-nums">
#{docChunkIndex} / {docTotalChunks - 1}
</span>
</div>
</div>
{/* Scrollable content */}
<div className="flex-1 overflow-y-auto min-h-0 p-4 space-y-3">
{/* Overlap from previous chunk */}
{prevChunk && (
<div className="text-xs text-slate-400 bg-amber-50 border-l-2 border-amber-300 px-3 py-2 rounded-r">
<div className="font-medium text-amber-600 mb-1">&#8593; Ende vorheriger Chunk #{docChunkIndex - 1}</div>
<p className="whitespace-pre-wrap break-words leading-relaxed">{getOverlapPrev()}</p>
</div>
)}
{/* Current chunk text */}
{currentChunk ? (
<div className="text-sm text-slate-800 whitespace-pre-wrap break-words leading-relaxed border-l-2 border-teal-400 pl-3">
{getChunkText(currentChunk)}
</div>
) : (
<div className="text-sm text-slate-400 italic">Kein Chunk-Text vorhanden.</div>
)}
{/* Overlap from next chunk */}
{nextChunk && (
<div className="text-xs text-slate-400 bg-amber-50 border-l-2 border-amber-300 px-3 py-2 rounded-r">
<div className="font-medium text-amber-600 mb-1">&#8595; Anfang naechster Chunk #{docChunkIndex + 1}</div>
<p className="whitespace-pre-wrap break-words leading-relaxed">{getOverlapNext()}</p>
</div>
)}
{/* Metadata */}
{currentChunk && (
<div className="mt-4 pt-3 border-t border-slate-100">
<div className="text-xs font-medium text-slate-500 mb-2">Metadaten</div>
<div className="grid grid-cols-2 gap-x-4 gap-y-1 text-xs">
{Object.entries(currentChunk)
.filter(([k]) => !HIDDEN_KEYS.has(k))
.sort(([a], [b]) => {
// Structural keys first
const aStruct = STRUCTURAL_KEYS.has(a) ? 0 : 1
const bStruct = STRUCTURAL_KEYS.has(b) ? 0 : 1
return aStruct - bStruct || a.localeCompare(b)
})
.map(([k, v]) => (
<div key={k} className={`flex gap-1 ${STRUCTURAL_KEYS.has(k) ? 'col-span-2 font-medium' : ''}`}>
<span className="font-medium text-slate-500 flex-shrink-0">{k}:</span>
<span className="text-slate-700 break-all">
{Array.isArray(v) ? v.join(', ') : String(v)}
</span>
</div>
))}
</div>
{/* Chunk quality indicator */}
<div className="mt-3 pt-2 border-t border-slate-50">
<div className="text-xs text-slate-400">
Chunk-Laenge: {getChunkText(currentChunk).length} Zeichen
{getChunkText(currentChunk).length < 50 && (
<span className="ml-2 text-orange-500 font-medium">&#9888; Sehr kurz</span>
)}
{getChunkText(currentChunk).length > 2000 && (
<span className="ml-2 text-orange-500 font-medium">&#9888; Sehr lang</span>
)}
</div>
</div>
</div>
)}
</div>
</div>
{/* PDF-Viewer Panel */}
{splitViewActive && (
<div className="bg-white rounded-xl border border-slate-200 flex flex-col min-h-0 overflow-hidden">
<div className="flex-shrink-0 px-4 py-2 bg-slate-50 border-b border-slate-100 flex items-center justify-between">
<span className="text-sm font-medium text-slate-700">Original-PDF</span>
<div className="flex items-center gap-2">
<span className="text-xs text-slate-400">
Seite ~{pdfPage}
{pdfMapping?.totalPages ? ` / ${pdfMapping.totalPages}` : ''}
</span>
{pdfUrl && (
<a
href={pdfUrl.split('#')[0]}
target="_blank"
rel="noopener noreferrer"
className="text-xs text-teal-600 hover:text-teal-800 underline"
>
Oeffnen &#8599;
</a>
)}
</div>
</div>
<div className="flex-1 min-h-0 relative">
{pdfUrl && pdfExists ? (
<iframe
key={`${selectedRegulation}-${pdfPage}`}
src={pdfUrl}
className="absolute inset-0 w-full h-full border-0"
title="Original PDF"
/>
) : (
<div className="flex items-center justify-center h-full text-slate-400 text-sm p-4">
<div className="text-center space-y-2">
<div className="text-3xl">&#128196;</div>
{!pdfMapping ? (
<>
<p>Kein PDF-Mapping fuer {selectedRegulation}.</p>
<p className="text-xs">rag-pdf-mapping.ts ergaenzen.</p>
</>
) : pdfExists === false ? (
<>
<p className="font-medium text-orange-600">PDF nicht vorhanden</p>
<p className="text-xs">Datei <code className="bg-slate-100 px-1 rounded">{pdfMapping.filename}</code> fehlt in ~/rag-originals/</p>
<p className="text-xs mt-1">Bitte manuell herunterladen und dort ablegen.</p>
</>
) : (
<p>PDF wird geprueft...</p>
)}
</div>
</div>
)}
</div>
</div>
)}
</div>
)}
</div>
</div>
)
}

View File

@@ -0,0 +1,126 @@
export interface RagPdfMapping {
filename: string
totalPages?: number
chunksPerPage?: number
language: string
}
export const RAG_PDF_MAPPING: Record<string, RagPdfMapping> = {
// EU Verordnungen
GDPR: { filename: 'GDPR_DE.pdf', language: 'de', totalPages: 88 },
EPRIVACY: { filename: 'EPRIVACY_DE.pdf', language: 'de' },
SCC: { filename: 'SCC_DE.pdf', language: 'de' },
SCC_FULL_TEXT: { filename: 'SCC_FULL_TEXT_DE.pdf', language: 'de' },
AIACT: { filename: 'AIACT_DE.pdf', language: 'de', totalPages: 144 },
CRA: { filename: 'CRA_DE.pdf', language: 'de' },
NIS2: { filename: 'NIS2_DE.pdf', language: 'de' },
DGA: { filename: 'DGA_DE.pdf', language: 'de' },
DSA: { filename: 'DSA_DE.pdf', language: 'de' },
PLD: { filename: 'PLD_DE.pdf', language: 'de' },
E_COMMERCE_RL: { filename: 'E_COMMERCE_RL_DE.pdf', language: 'de' },
VERBRAUCHERRECHTE_RL: { filename: 'VERBRAUCHERRECHTE_RL_DE.pdf', language: 'de' },
DIGITALE_INHALTE_RL: { filename: 'DIGITALE_INHALTE_RL_DE.pdf', language: 'de' },
DMA: { filename: 'DMA_DE.pdf', language: 'de' },
DPF: { filename: 'DPF_DE.pdf', language: 'de' },
EUCSA: { filename: 'EUCSA_DE.pdf', language: 'de' },
DATAACT: { filename: 'DATAACT_DE.pdf', language: 'de' },
DORA: { filename: 'DORA_DE.pdf', language: 'de' },
PSD2: { filename: 'PSD2_DE.pdf', language: 'de' },
AMLR: { filename: 'AMLR_DE.pdf', language: 'de' },
MiCA: { filename: 'MiCA_DE.pdf', language: 'de' },
EHDS: { filename: 'EHDS_DE.pdf', language: 'de' },
EAA: { filename: 'EAA_DE.pdf', language: 'de' },
DSM: { filename: 'DSM_DE.pdf', language: 'de' },
GPSR: { filename: 'GPSR_DE.pdf', language: 'de' },
MACHINERY_REG: { filename: 'MACHINERY_REG_DE.pdf', language: 'de' },
BLUE_GUIDE: { filename: 'BLUE_GUIDE_DE.pdf', language: 'de' },
// DE Gesetze
TDDDG: { filename: 'TDDDG_DE.pdf', language: 'de' },
BDSG_FULL: { filename: 'BDSG_FULL_DE.pdf', language: 'de' },
DE_DDG: { filename: 'DE_DDG.pdf', language: 'de' },
DE_BGB_AGB: { filename: 'DE_BGB_AGB.pdf', language: 'de' },
DE_EGBGB: { filename: 'DE_EGBGB.pdf', language: 'de' },
DE_HGB_RET: { filename: 'DE_HGB_RET.pdf', language: 'de' },
DE_AO_RET: { filename: 'DE_AO_RET.pdf', language: 'de' },
DE_UWG: { filename: 'DE_UWG.pdf', language: 'de' },
DE_TKG: { filename: 'DE_TKG.pdf', language: 'de' },
DE_PANGV: { filename: 'DE_PANGV.pdf', language: 'de' },
DE_DLINFOV: { filename: 'DE_DLINFOV.pdf', language: 'de' },
DE_BETRVG: { filename: 'DE_BETRVG.pdf', language: 'de' },
DE_GESCHGEHG: { filename: 'DE_GESCHGEHG.pdf', language: 'de' },
DE_BSIG: { filename: 'DE_BSIG.pdf', language: 'de' },
DE_USTG_RET: { filename: 'DE_USTG_RET.pdf', language: 'de' },
// BSI Standards
'BSI-TR-03161-1': { filename: 'BSI-TR-03161-1.pdf', language: 'de' },
'BSI-TR-03161-2': { filename: 'BSI-TR-03161-2.pdf', language: 'de' },
'BSI-TR-03161-3': { filename: 'BSI-TR-03161-3.pdf', language: 'de' },
// AT Gesetze
AT_DSG: { filename: 'AT_DSG.pdf', language: 'de' },
AT_DSG_FULL: { filename: 'AT_DSG_FULL.pdf', language: 'de' },
AT_ECG: { filename: 'AT_ECG.pdf', language: 'de' },
AT_TKG: { filename: 'AT_TKG.pdf', language: 'de' },
AT_KSCHG: { filename: 'AT_KSCHG.pdf', language: 'de' },
AT_FAGG: { filename: 'AT_FAGG.pdf', language: 'de' },
AT_UGB_RET: { filename: 'AT_UGB_RET.pdf', language: 'de' },
AT_BAO_RET: { filename: 'AT_BAO_RET.pdf', language: 'de' },
AT_MEDIENG: { filename: 'AT_MEDIENG.pdf', language: 'de' },
AT_ABGB_AGB: { filename: 'AT_ABGB_AGB.pdf', language: 'de' },
AT_UWG: { filename: 'AT_UWG.pdf', language: 'de' },
// CH Gesetze
CH_DSG: { filename: 'CH_DSG.pdf', language: 'de' },
CH_DSV: { filename: 'CH_DSV.pdf', language: 'de' },
CH_OR_AGB: { filename: 'CH_OR_AGB.pdf', language: 'de' },
CH_UWG: { filename: 'CH_UWG.pdf', language: 'de' },
CH_FMG: { filename: 'CH_FMG.pdf', language: 'de' },
CH_GEBUV: { filename: 'CH_GEBUV.pdf', language: 'de' },
CH_ZERTES: { filename: 'CH_ZERTES.pdf', language: 'de' },
CH_ZGB_PERS: { filename: 'CH_ZGB_PERS.pdf', language: 'de' },
// LI
LI_DSG: { filename: 'LI_DSG.pdf', language: 'de' },
// Nationale DSG (andere EU)
ES_LOPDGDD: { filename: 'ES_LOPDGDD.pdf', language: 'es' },
IT_CODICE_PRIVACY: { filename: 'IT_CODICE_PRIVACY.pdf', language: 'it' },
NL_UAVG: { filename: 'NL_UAVG.pdf', language: 'nl' },
FR_CNIL_GUIDE: { filename: 'FR_CNIL_GUIDE.pdf', language: 'fr' },
IE_DPA_2018: { filename: 'IE_DPA_2018.pdf', language: 'en' },
UK_DPA_2018: { filename: 'UK_DPA_2018.pdf', language: 'en' },
UK_GDPR: { filename: 'UK_GDPR.pdf', language: 'en' },
NO_PERSONOPPLYSNINGSLOVEN: { filename: 'NO_PERSONOPPLYSNINGSLOVEN.pdf', language: 'no' },
SE_DATASKYDDSLAG: { filename: 'SE_DATASKYDDSLAG.pdf', language: 'sv' },
PL_UODO: { filename: 'PL_UODO.pdf', language: 'pl' },
CZ_ZOU: { filename: 'CZ_ZOU.pdf', language: 'cs' },
HU_INFOTV: { filename: 'HU_INFOTV.pdf', language: 'hu' },
BE_DPA_LAW: { filename: 'BE_DPA_LAW.pdf', language: 'nl' },
FI_TIETOSUOJALAKI: { filename: 'FI_TIETOSUOJALAKI.pdf', language: 'fi' },
DK_DATABESKYTTELSESLOVEN: { filename: 'DK_DATABESKYTTELSESLOVEN.pdf', language: 'da' },
LU_DPA_LAW: { filename: 'LU_DPA_LAW.pdf', language: 'fr' },
// DE Gesetze (zusaetzlich)
TMG_KOMPLETT: { filename: 'TMG_KOMPLETT.pdf', language: 'de' },
DE_URHG: { filename: 'DE_URHG.pdf', language: 'de' },
// EDPB Guidelines
EDPB_GUIDELINES_5_2020: { filename: 'EDPB_GUIDELINES_5_2020.pdf', language: 'en' },
EDPB_GUIDELINES_7_2020: { filename: 'EDPB_GUIDELINES_7_2020.pdf', language: 'en' },
EDPB_GUIDELINES_1_2020: { filename: 'EDPB_GUIDELINES_1_2020.pdf', language: 'en' },
EDPB_GUIDELINES_1_2022: { filename: 'EDPB_GUIDELINES_1_2022.pdf', language: 'en' },
EDPB_GUIDELINES_2_2023: { filename: 'EDPB_GUIDELINES_2_2023.pdf', language: 'en' },
EDPB_GUIDELINES_2_2024: { filename: 'EDPB_GUIDELINES_2_2024.pdf', language: 'en' },
EDPB_GUIDELINES_4_2019: { filename: 'EDPB_GUIDELINES_4_2019.pdf', language: 'en' },
EDPB_GUIDELINES_9_2022: { filename: 'EDPB_GUIDELINES_9_2022.pdf', language: 'en' },
EDPB_DPIA_LIST: { filename: 'EDPB_DPIA_LIST.pdf', language: 'en' },
EDPB_LEGITIMATE_INTEREST: { filename: 'EDPB_LEGITIMATE_INTEREST.pdf', language: 'en' },
// EDPS
EDPS_DPIA_LIST: { filename: 'EDPS_DPIA_LIST.pdf', language: 'en' },
// Frameworks
ENISA_SECURE_BY_DESIGN: { filename: 'ENISA_SECURE_BY_DESIGN.pdf', language: 'en' },
ENISA_SUPPLY_CHAIN: { filename: 'ENISA_SUPPLY_CHAIN.pdf', language: 'en' },
ENISA_THREAT_LANDSCAPE: { filename: 'ENISA_THREAT_LANDSCAPE.pdf', language: 'en' },
ENISA_ICS_SCADA: { filename: 'ENISA_ICS_SCADA.pdf', language: 'en' },
ENISA_CYBERSECURITY_2024: { filename: 'ENISA_CYBERSECURITY_2024.pdf', language: 'en' },
NIST_SSDF: { filename: 'NIST_SSDF.pdf', language: 'en' },
NIST_CSF_2: { filename: 'NIST_CSF_2.pdf', language: 'en' },
OECD_AI_PRINCIPLES: { filename: 'OECD_AI_PRINCIPLES.pdf', language: 'en' },
// EU-IFRS / EFRAG
EU_IFRS_DE: { filename: 'EU_IFRS_DE.pdf', language: 'de' },
EU_IFRS_EN: { filename: 'EU_IFRS_EN.pdf', language: 'en' },
EFRAG_ENDORSEMENT: { filename: 'EFRAG_ENDORSEMENT.pdf', language: 'en' },
}

View File

@@ -11,6 +11,8 @@ import React, { useState, useEffect, useCallback } from 'react'
import Link from 'next/link'
import { PagePurpose } from '@/components/common/PagePurpose'
import { AIModuleSidebarResponsive } from '@/components/ai/AIModuleSidebar'
import { REGULATIONS_IN_RAG } from './rag-constants'
import { ChunkBrowserQA } from './components/ChunkBrowserQA'
// API uses local proxy route to klausur-service
const API_PROXY = '/api/legal-corpus'
@@ -73,7 +75,7 @@ interface DsfaCorpusStatus {
type RegulationCategory = 'regulations' | 'dsfa' | 'nibis' | 'templates'
// Tab definitions
type TabId = 'overview' | 'regulations' | 'map' | 'search' | 'data' | 'ingestion' | 'pipeline'
type TabId = 'overview' | 'regulations' | 'map' | 'search' | 'chunks' | 'data' | 'ingestion' | 'pipeline'
// Custom document type
interface CustomDocument {
@@ -1011,8 +1013,264 @@ const REGULATIONS = [
keyTopics: ['Bussgeldberechnung', 'Schweregrad', 'Milderungsgruende', 'Bussgeldrahmen'],
effectiveDate: '2022'
},
// =====================================================================
// Neu ingestierte EU-Richtlinien (Februar 2026)
// =====================================================================
{
code: 'E_COMMERCE_RL',
name: 'E-Commerce-Richtlinie',
fullName: 'Richtlinie 2000/31/EG ueber den elektronischen Geschaeftsverkehr',
type: 'eu_directive',
expected: 30,
description: 'EU-Richtlinie ueber den elektronischen Geschaeftsverkehr (E-Commerce). Regelt Herkunftslandprinzip, Informationspflichten, Haftungsprivilegien fuer Vermittler (Mere Conduit, Caching, Hosting).',
relevantFor: ['Online-Dienste', 'E-Commerce', 'Hosting-Anbieter', 'Plattformen'],
keyTopics: ['Herkunftslandprinzip', 'Haftungsprivileg', 'Informationspflichten', 'Spam-Verbot', 'Vermittlerhaftung'],
effectiveDate: '17. Juli 2000'
},
{
code: 'VERBRAUCHERRECHTE_RL',
name: 'Verbraucherrechte-Richtlinie',
fullName: 'Richtlinie 2011/83/EU ueber die Rechte der Verbraucher',
type: 'eu_directive',
expected: 25,
description: 'EU-weite Harmonisierung der Verbraucherrechte bei Fernabsatz und aussergeschaeftlichen Vertraegen. 14-Tage-Widerrufsrecht, Informationspflichten, digitale Inhalte.',
relevantFor: ['Online-Shops', 'E-Commerce', 'Fernabsatz', 'Dienstleister'],
keyTopics: ['Widerrufsrecht 14 Tage', 'Informationspflichten', 'Fernabsatzvertraege', 'Digitale Inhalte'],
effectiveDate: '13. Juni 2014'
},
{
code: 'DIGITALE_INHALTE_RL',
name: 'Digitale-Inhalte-Richtlinie',
fullName: 'Richtlinie (EU) 2019/770 ueber digitale Inhalte und Dienstleistungen',
type: 'eu_directive',
expected: 20,
description: 'Gewaehrleistungsrecht fuer digitale Inhalte und Dienstleistungen. Regelt Maengelhaftung, Updates, Vertragsmaessigkeit und Kuendigungsrechte bei digitalen Produkten.',
relevantFor: ['SaaS-Anbieter', 'App-Entwickler', 'Cloud-Dienste', 'Streaming-Anbieter', 'Software-Hersteller'],
keyTopics: ['Digitale Gewaehrleistung', 'Update-Pflicht', 'Vertragsmaessigkeit', 'Kuendigungsrecht', 'Datenportabilitaet'],
effectiveDate: '1. Januar 2022'
},
{
code: 'DMA',
name: 'Digital Markets Act',
fullName: 'Verordnung (EU) 2022/1925 - Digital Markets Act',
type: 'eu_regulation',
expected: 50,
description: 'Reguliert digitale Gatekeeper-Plattformen. Stellt Verhaltensregeln fuer grosse Plattformen auf (Apple, Google, Meta, Amazon, Microsoft). Verbietet Selbstbevorzugung und erzwingt Interoperabilitaet.',
relevantFor: ['Grosse Plattformen', 'App-Stores', 'Suchmaschinen', 'Social Media', 'Messenger-Dienste'],
keyTopics: ['Gatekeeper-Pflichten', 'Interoperabilitaet', 'Selbstbevorzugung', 'App-Store-Regeln', 'Datenportabilitaet'],
effectiveDate: '2. Mai 2023'
},
// === Industrie-Compliance (2026-02-28) ===
{
code: 'MACHINERY_REG',
name: 'Maschinenverordnung',
fullName: 'Verordnung (EU) 2023/1230 ueber Maschinen (Machinery Regulation)',
type: 'eu_regulation',
expected: 100,
description: 'Loest die alte Maschinenrichtlinie 2006/42/EG ab. Regelt Sicherheitsanforderungen fuer Maschinen und zugehoerige Produkte, CE-Kennzeichnung, Konformitaetsbewertung und Marktaufsicht. Neu: Cybersecurity-Anforderungen fuer vernetzte Maschinen.',
relevantFor: ['Maschinenbau', 'Industrie 4.0', 'Automatisierung', 'Hersteller', 'Importeure'],
keyTopics: ['CE-Kennzeichnung', 'Konformitaetsbewertung', 'Risikobeurteilung', 'Cybersecurity', 'Betriebsanleitung'],
effectiveDate: '20. Januar 2027'
},
{
code: 'BLUE_GUIDE',
name: 'Blue Guide',
fullName: 'Leitfaden fuer die Umsetzung der EU-Produktvorschriften (Blue Guide 2022)',
type: 'eu_guideline',
expected: 200,
description: 'Umfassender Leitfaden der EU-Kommission zur Umsetzung von Produktvorschriften. Erklaert CE-Kennzeichnung, Konformitaetsbewertungsverfahren, notifizierte Stellen, Marktaufsicht und den New Legislative Framework.',
relevantFor: ['Hersteller', 'Importeure', 'Haendler', 'Notifizierte Stellen', 'Marktaufsichtsbehoerden'],
keyTopics: ['CE-Kennzeichnung', 'Konformitaetserklaerung', 'Notifizierte Stellen', 'Marktaufsicht', 'New Legislative Framework'],
effectiveDate: '29. Juni 2022'
},
{
code: 'ENISA_SECURE_BY_DESIGN',
name: 'ENISA Secure by Design',
fullName: 'ENISA Secure Software Development Best Practices',
type: 'eu_guideline',
expected: 50,
description: 'ENISA-Leitfaden fuer sichere Softwareentwicklung. Beschreibt Best Practices fuer Security by Design, sichere Entwicklungsprozesse und Schwachstellenmanagement.',
relevantFor: ['Softwareentwickler', 'DevOps', 'IT-Sicherheit', 'Produktmanagement'],
keyTopics: ['Security by Design', 'SDLC', 'Schwachstellenmanagement', 'Secure Coding', 'Threat Modeling'],
effectiveDate: '2023'
},
{
code: 'ENISA_SUPPLY_CHAIN',
name: 'ENISA Supply Chain Security',
fullName: 'ENISA Threat Landscape for Supply Chain Attacks',
type: 'eu_guideline',
expected: 50,
description: 'ENISA-Analyse der Bedrohungslandschaft fuer Supply-Chain-Angriffe. Beschreibt Angriffsvektoren, Taxonomie und Empfehlungen zur Absicherung von Software-Lieferketten.',
relevantFor: ['IT-Sicherheit', 'Beschaffung', 'Softwareentwickler', 'CISO'],
keyTopics: ['Supply Chain Security', 'SolarWinds', 'SBOM', 'Lieferantenrisiko', 'Third-Party Risk'],
effectiveDate: '2021'
},
{
code: 'NIST_SSDF',
name: 'NIST SSDF',
fullName: 'NIST SP 800-218 — Secure Software Development Framework (SSDF)',
type: 'international_standard',
expected: 40,
description: 'NIST-Framework fuer sichere Softwareentwicklung. Definiert Praktiken und Aufgaben in vier Gruppen: Prepare, Protect, Produce, Respond. Weit verbreitet als Referenz fuer Software Supply Chain Security.',
relevantFor: ['Softwareentwickler', 'DevSecOps', 'IT-Sicherheit', 'Compliance-Manager'],
keyTopics: ['SSDF', 'Secure SDLC', 'Software Supply Chain', 'Vulnerability Management', 'Code Review'],
effectiveDate: '3. Februar 2022'
},
{
code: 'NIST_CSF_2',
name: 'NIST CSF 2.0',
fullName: 'NIST Cybersecurity Framework (CSF) 2.0',
type: 'international_standard',
expected: 50,
description: 'Version 2.0 des NIST Cybersecurity Framework. Neue Kernfunktion "Govern" ergaenzt Identify, Protect, Detect, Respond, Recover. Erweitert den Anwendungsbereich ueber kritische Infrastruktur hinaus auf alle Organisationen.',
relevantFor: ['CISO', 'IT-Sicherheit', 'Risikomanagement', 'Geschaeftsfuehrung', 'Alle Branchen'],
keyTopics: ['Govern', 'Identify', 'Protect', 'Detect', 'Respond', 'Recover', 'Cybersecurity'],
effectiveDate: '26. Februar 2024'
},
{
code: 'OECD_AI_PRINCIPLES',
name: 'OECD AI Principles',
fullName: 'OECD Recommendation on Artificial Intelligence (AI Principles)',
type: 'international_standard',
expected: 20,
description: 'OECD-Empfehlung zu Kuenstlicher Intelligenz. Definiert fuenf Prinzipien fuer verantwortungsvolle KI: Inklusives Wachstum, Menschenzentrierte Werte, Transparenz, Robustheit und Rechenschaftspflicht. Von 46 Laendern angenommen.',
relevantFor: ['KI-Entwickler', 'Policy-Maker', 'Ethik-Kommissionen', 'Geschaeftsfuehrung'],
keyTopics: ['AI Ethics', 'Transparenz', 'Accountability', 'Trustworthy AI', 'Human-Centered AI'],
effectiveDate: '22. Mai 2019'
},
{
code: 'EU_IFRS',
name: 'EU-IFRS',
fullName: 'Verordnung (EU) 2023/1803 — International Financial Reporting Standards',
type: 'eu_regulation',
expected: 500,
description: 'Konsolidierte Fassung der von der EU uebernommenen IFRS/IAS/IFRIC/SIC. Rechtsverbindlich fuer boersennotierte EU-Unternehmen. Enthalt IFRS 1-17, IAS 1-41, IFRIC 1-23 und SIC 7-32 in der EU-endorsed Fassung (Stand Okt 2023). ACHTUNG: Neuere IASB-Standards sind moeglicherweise noch nicht EU-endorsed.',
relevantFor: ['Rechnungswesen', 'Wirtschaftspruefer', 'boersennotierte Unternehmen', 'Finanzberichterstattung', 'CFO'],
keyTopics: ['IFRS 16 Leasing', 'IFRS 9 Finanzinstrumente', 'IAS 1 Darstellung', 'IFRS 15 Erloese', 'IFRS 17 Versicherungsvertraege', 'Konsolidierung'],
effectiveDate: '16. Oktober 2023'
},
{
code: 'EFRAG_ENDORSEMENT',
name: 'EFRAG Endorsement Status',
fullName: 'EFRAG EU Endorsement Status Report (Dezember 2025)',
type: 'eu_guideline',
expected: 30,
description: 'Uebersicht des European Financial Reporting Advisory Group (EFRAG) ueber den EU-Endorsement-Stand aller IFRS/IAS-Standards. Zeigt welche Standards von der EU uebernommen wurden und welche noch ausstehend sind. Relevant fuer internationale Ausschreibungen und Compliance-Pruefung.',
relevantFor: ['Rechnungswesen', 'Wirtschaftspruefer', 'Compliance Officer', 'internationale Ausschreibungen'],
keyTopics: ['EU Endorsement', 'IFRS 18', 'IFRS S1/S2 Sustainability', 'Endorsement Status', 'IASB Updates'],
effectiveDate: '18. Dezember 2025'
},
]
// Source URLs for original documents (click to view original)
const REGULATION_SOURCES: Record<string, string> = {
// EU Verordnungen/Richtlinien (EUR-Lex)
GDPR: 'https://eur-lex.europa.eu/legal-content/DE/TXT/?uri=CELEX:32016R0679',
EPRIVACY: 'https://eur-lex.europa.eu/legal-content/DE/TXT/?uri=CELEX:32002L0058',
SCC: 'https://eur-lex.europa.eu/legal-content/DE/TXT/?uri=CELEX:32021D0914',
DPF: 'https://eur-lex.europa.eu/legal-content/DE/TXT/?uri=CELEX:32023D1795',
AIACT: 'https://eur-lex.europa.eu/legal-content/DE/TXT/?uri=CELEX:32024R1689',
CRA: 'https://eur-lex.europa.eu/legal-content/DE/TXT/?uri=CELEX:32024R2847',
NIS2: 'https://eur-lex.europa.eu/legal-content/DE/TXT/?uri=CELEX:32022L2555',
EUCSA: 'https://eur-lex.europa.eu/legal-content/DE/TXT/?uri=CELEX:32019R0881',
DATAACT: 'https://eur-lex.europa.eu/legal-content/DE/TXT/?uri=CELEX:32023R2854',
DGA: 'https://eur-lex.europa.eu/legal-content/DE/TXT/?uri=CELEX:32022R0868',
DSA: 'https://eur-lex.europa.eu/legal-content/DE/TXT/?uri=CELEX:32022R2065',
EAA: 'https://eur-lex.europa.eu/legal-content/DE/TXT/?uri=CELEX:32019L0882',
DSM: 'https://eur-lex.europa.eu/legal-content/DE/TXT/?uri=CELEX:32019L0790',
PLD: 'https://eur-lex.europa.eu/legal-content/DE/TXT/?uri=CELEX:32024L2853',
GPSR: 'https://eur-lex.europa.eu/legal-content/DE/TXT/?uri=CELEX:32023R0988',
DORA: 'https://eur-lex.europa.eu/legal-content/DE/TXT/?uri=CELEX:32022R2554',
PSD2: 'https://eur-lex.europa.eu/legal-content/DE/TXT/?uri=CELEX:32015L2366',
AMLR: 'https://eur-lex.europa.eu/legal-content/DE/TXT/?uri=CELEX:32024R1624',
MiCA: 'https://eur-lex.europa.eu/legal-content/DE/TXT/?uri=CELEX:32023R1114',
EHDS: 'https://eur-lex.europa.eu/legal-content/DE/TXT/?uri=CELEX:32025R0327',
SCC_FULL_TEXT: 'https://eur-lex.europa.eu/legal-content/DE/TXT/?uri=CELEX:32021D0914',
E_COMMERCE_RL: 'https://eur-lex.europa.eu/legal-content/DE/TXT/?uri=CELEX:32000L0031',
VERBRAUCHERRECHTE_RL: 'https://eur-lex.europa.eu/legal-content/DE/TXT/?uri=CELEX:32011L0083',
DIGITALE_INHALTE_RL: 'https://eur-lex.europa.eu/legal-content/DE/TXT/?uri=CELEX:32019L0770',
DMA: 'https://eur-lex.europa.eu/legal-content/DE/TXT/?uri=CELEX:32022R1925',
MACHINERY_REG: 'https://eur-lex.europa.eu/legal-content/DE/TXT/?uri=CELEX:32023R1230',
BLUE_GUIDE: 'https://eur-lex.europa.eu/legal-content/DE/TXT/?uri=CELEX:52022XC0629(04)',
EU_IFRS: 'https://eur-lex.europa.eu/legal-content/DE/TXT/?uri=CELEX:32023R1803',
// EDPB Guidelines
EDPB_GUIDELINES_2_2019: 'https://www.edpb.europa.eu/our-work-tools/our-documents/guidelines/guidelines-22019-processing-personal-data-under-article-61b_en',
EDPB_GUIDELINES_3_2019: 'https://www.edpb.europa.eu/our-work-tools/our-documents/guidelines/guidelines-32019-processing-personal-data-through-video_en',
EDPB_GUIDELINES_5_2020: 'https://www.edpb.europa.eu/our-work-tools/our-documents/guidelines/guidelines-052020-consent-under-regulation-2016679_en',
EDPB_GUIDELINES_7_2020: 'https://www.edpb.europa.eu/our-work-tools/our-documents/guidelines/guidelines-072020-concepts-controller-and-processor-gdpr_en',
EDPB_GUIDELINES_1_2022: 'https://www.edpb.europa.eu/our-work-tools/our-documents/guidelines/guidelines-042022-calculation-administrative-fines-under-gdpr_en',
// BSI Technische Richtlinien
'BSI-TR-03161-1': 'https://www.bsi.bund.de/SharedDocs/Downloads/DE/BSI/Publikationen/TechnischeRichtlinien/TR03161/BSI-TR-03161-1.html',
'BSI-TR-03161-2': 'https://www.bsi.bund.de/SharedDocs/Downloads/DE/BSI/Publikationen/TechnischeRichtlinien/TR03161/BSI-TR-03161-2.html',
'BSI-TR-03161-3': 'https://www.bsi.bund.de/SharedDocs/Downloads/DE/BSI/Publikationen/TechnischeRichtlinien/TR03161/BSI-TR-03161-3.html',
// Nationale Datenschutzgesetze
AT_DSG: 'https://www.ris.bka.gv.at/GeltendeFassung.wxe?Abfrage=Bundesnormen&Gesetzesnummer=10001597',
BDSG_FULL: 'https://www.gesetze-im-internet.de/bdsg_2018/',
CH_DSG: 'https://www.fedlex.admin.ch/eli/cc/2022/491/de',
LI_DSG: 'https://www.gesetze.li/konso/2018.272',
BE_DPA_LAW: 'https://www.autoriteprotectiondonnees.be/citoyen/la-loi-du-30-juillet-2018',
NL_UAVG: 'https://wetten.overheid.nl/BWBR0040940/',
FR_CNIL_GUIDE: 'https://www.cnil.fr/fr/rgpd-par-ou-commencer',
ES_LOPDGDD: 'https://www.boe.es/buscar/act.php?id=BOE-A-2018-16673',
IT_CODICE_PRIVACY: 'https://www.garanteprivacy.it/home/docweb/-/docweb-display/docweb/9042678',
IE_DPA_2018: 'https://www.irishstatutebook.ie/eli/2018/act/7/enacted/en/html',
UK_DPA_2018: 'https://www.legislation.gov.uk/ukpga/2018/12/contents',
UK_GDPR: 'https://www.legislation.gov.uk/eur/2016/679/contents',
NO_PERSONOPPLYSNINGSLOVEN: 'https://lovdata.no/dokument/NL/lov/2018-06-15-38',
SE_DATASKYDDSLAG: 'https://www.riksdagen.se/sv/dokument-och-lagar/dokument/svensk-forfattningssamling/lag-2018218-med-kompletterande-bestammelser_sfs-2018-218/',
FI_TIETOSUOJALAKI: 'https://www.finlex.fi/fi/laki/ajantasa/2018/20181050',
PL_UODO: 'https://isap.sejm.gov.pl/isap.nsf/DocDetails.xsp?id=WDU20180001000',
CZ_ZOU: 'https://www.zakonyprolidi.cz/cs/2019-110',
HU_INFOTV: 'https://net.jogtar.hu/jogszabaly?docid=a1100112.tv',
LU_DPA_LAW: 'https://legilux.public.lu/eli/etat/leg/loi/2018/08/01/a686/jo',
DK_DATABESKYTTELSESLOVEN: 'https://www.retsinformation.dk/eli/lta/2018/502',
// Deutschland — Weitere Gesetze
TDDDG: 'https://www.gesetze-im-internet.de/tdddg/',
DE_DDG: 'https://www.gesetze-im-internet.de/ddg/',
DE_BGB_AGB: 'https://www.gesetze-im-internet.de/bgb/__305.html',
DE_EGBGB: 'https://www.gesetze-im-internet.de/bgbeg/art_246.html',
DE_UWG: 'https://www.gesetze-im-internet.de/uwg_2004/',
DE_HGB_RET: 'https://www.gesetze-im-internet.de/hgb/__257.html',
DE_AO_RET: 'https://www.gesetze-im-internet.de/ao_1977/__147.html',
DE_TKG: 'https://www.gesetze-im-internet.de/tkg_2021/',
DE_PANGV: 'https://www.gesetze-im-internet.de/pangv_2022/',
DE_DLINFOV: 'https://www.gesetze-im-internet.de/dlinfov/',
DE_BETRVG: 'https://www.gesetze-im-internet.de/betrvg/__87.html',
DE_GESCHGEHG: 'https://www.gesetze-im-internet.de/geschgehg/',
DE_BSIG: 'https://www.gesetze-im-internet.de/bsig_2009/',
DE_USTG_RET: 'https://www.gesetze-im-internet.de/ustg_1980/__14b.html',
// Oesterreich — Weitere Gesetze
AT_ECG: 'https://www.ris.bka.gv.at/GeltendeFassung.wxe?Abfrage=Bundesnormen&Gesetzesnummer=20001703',
AT_TKG: 'https://www.ris.bka.gv.at/GeltendeFassung.wxe?Abfrage=Bundesnormen&Gesetzesnummer=20007898',
AT_KSCHG: 'https://www.ris.bka.gv.at/GeltendeFassung.wxe?Abfrage=Bundesnormen&Gesetzesnummer=10002462',
AT_FAGG: 'https://www.ris.bka.gv.at/GeltendeFassung.wxe?Abfrage=Bundesnormen&Gesetzesnummer=20008783',
AT_UGB_RET: 'https://www.ris.bka.gv.at/GeltendeFassung.wxe?Abfrage=Bundesnormen&Gesetzesnummer=10001702',
AT_BAO_RET: 'https://www.ris.bka.gv.at/GeltendeFassung.wxe?Abfrage=Bundesnormen&Gesetzesnummer=10003940',
AT_MEDIENG: 'https://www.ris.bka.gv.at/GeltendeFassung.wxe?Abfrage=Bundesnormen&Gesetzesnummer=10000719',
AT_ABGB_AGB: 'https://www.ris.bka.gv.at/GeltendeFassung.wxe?Abfrage=Bundesnormen&Gesetzesnummer=10001622',
AT_UWG: 'https://www.ris.bka.gv.at/GeltendeFassung.wxe?Abfrage=Bundesnormen&Gesetzesnummer=10002665',
// Schweiz
CH_DSV: 'https://www.fedlex.admin.ch/eli/cc/2022/568/de',
CH_OR_AGB: 'https://www.fedlex.admin.ch/eli/cc/27/317_321_377/de',
CH_UWG: 'https://www.fedlex.admin.ch/eli/cc/1988/223_223_223/de',
CH_FMG: 'https://www.fedlex.admin.ch/eli/cc/1997/2187_2187_2187/de',
CH_GEBUV: 'https://www.fedlex.admin.ch/eli/cc/2002/249/de',
CH_ZERTES: 'https://www.fedlex.admin.ch/eli/cc/2016/752/de',
CH_ZGB_PERS: 'https://www.fedlex.admin.ch/eli/cc/24/233_245_233/de',
// Industrie-Compliance
ENISA_SECURE_BY_DESIGN: 'https://www.enisa.europa.eu/publications/secure-development-best-practices',
ENISA_SUPPLY_CHAIN: 'https://www.enisa.europa.eu/publications/threat-landscape-for-supply-chain-attacks',
NIST_SSDF: 'https://csrc.nist.gov/pubs/sp/800/218/final',
NIST_CSF_2: 'https://www.nist.gov/cyberframework',
OECD_AI_PRINCIPLES: 'https://legalinstruments.oecd.org/en/instruments/OECD-LEGAL-0449',
// IFRS / EFRAG
EU_IFRS_DE: 'https://eur-lex.europa.eu/legal-content/DE/TXT/?uri=CELEX:32023R1803',
EU_IFRS_EN: 'https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32023R1803',
EFRAG_ENDORSEMENT: 'https://www.efrag.org/activities/endorsement-status-report',
// Full-text Datenschutzgesetz AT
AT_DSG_FULL: 'https://www.ris.bka.gv.at/GeltendeFassung.wxe?Abfrage=Bundesnormen&Gesetzesnummer=10001597',
}
// License info for each regulation
const REGULATION_LICENSES: Record<string, { license: string; licenseNote: string }> = {
GDPR: { license: 'PUBLIC_DOMAIN', licenseNote: 'Amtliches Werk der EU — frei verwendbar' },
@@ -1063,6 +1321,18 @@ const REGULATION_LICENSES: Record<string, { license: string; licenseNote: string
EDPB_GUIDELINES_3_2019: { license: 'EDPB-LICENSE', licenseNote: 'EDPB Document License' },
EDPB_GUIDELINES_5_2020: { license: 'EDPB-LICENSE', licenseNote: 'EDPB Document License' },
EDPB_GUIDELINES_7_2020: { license: 'EDPB-LICENSE', licenseNote: 'EDPB Document License' },
// Industrie-Compliance (2026-02-28)
MACHINERY_REG: { license: 'PUBLIC_DOMAIN', licenseNote: 'EU-Verordnung — amtliches Werk' },
BLUE_GUIDE: { license: 'PUBLIC_DOMAIN', licenseNote: 'EU-Leitfaden — amtliches Werk der Kommission' },
ENISA_SECURE_BY_DESIGN: { license: 'CC-BY-4.0', licenseNote: 'ENISA Publication — CC BY 4.0' },
ENISA_SUPPLY_CHAIN: { license: 'CC-BY-4.0', licenseNote: 'ENISA Publication — CC BY 4.0' },
NIST_SSDF: { license: 'PUBLIC_DOMAIN', licenseNote: 'US Government Work — Public Domain' },
NIST_CSF_2: { license: 'PUBLIC_DOMAIN', licenseNote: 'US Government Work — Public Domain' },
OECD_AI_PRINCIPLES: { license: 'PUBLIC_DOMAIN', licenseNote: 'OECD Legal Instrument — Reuse Notice' },
// EU-IFRS / EFRAG (2026-02-28)
EU_IFRS_DE: { license: 'PUBLIC_DOMAIN', licenseNote: 'EU-Verordnung — amtliches Werk' },
EU_IFRS_EN: { license: 'PUBLIC_DOMAIN', licenseNote: 'EU-Verordnung — amtliches Werk' },
EFRAG_ENDORSEMENT: { license: 'PUBLIC_DOMAIN', licenseNote: 'EFRAG — oeffentliches Dokument' },
// DACH National Laws — Deutschland
DE_DDG: { license: 'PUBLIC_DOMAIN', licenseNote: 'Deutsches Bundesgesetz — amtliches Werk (§5 UrhG)' },
DE_BGB_AGB: { license: 'PUBLIC_DOMAIN', licenseNote: 'Deutsches Bundesgesetz — amtliches Werk (§5 UrhG)' },
@@ -1099,6 +1369,35 @@ const REGULATION_LICENSES: Record<string, { license: string; licenseNote: string
LU_DPA_LAW: { license: 'PUBLIC_DOMAIN', licenseNote: 'Amtliches Werk Luxemburg — frei verwendbar' },
DK_DATABESKYTTELSESLOVEN: { license: 'PUBLIC_DOMAIN', licenseNote: 'Amtliches Werk Daenemark — frei verwendbar' },
EDPB_GUIDELINES_1_2022: { license: 'EDPB-LICENSE', licenseNote: 'EDPB Document License' },
// Neue EU-Richtlinien (Februar 2026 ingestiert)
E_COMMERCE_RL: { license: 'PUBLIC_DOMAIN', licenseNote: 'EU-Richtlinie — amtliches Werk' },
VERBRAUCHERRECHTE_RL: { license: 'PUBLIC_DOMAIN', licenseNote: 'EU-Richtlinie — amtliches Werk' },
DIGITALE_INHALTE_RL: { license: 'PUBLIC_DOMAIN', licenseNote: 'EU-Richtlinie — amtliches Werk' },
DMA: { license: 'PUBLIC_DOMAIN', licenseNote: 'EU-Verordnung — amtliches Werk' },
}
// REGULATIONS_IN_RAG is imported from ./rag-constants.ts
// Helper: Check if regulation is in RAG
const isInRag = (code: string): boolean => code in REGULATIONS_IN_RAG
// Helper: Get known chunk count for a regulation
const getKnownChunks = (code: string): number => REGULATIONS_IN_RAG[code]?.chunks || 0
// Known collection totals (updated: 2026-02-28)
// Note: bp_compliance_ce and bp_compliance_datenschutz will increase after
// industry compliance ingestion (Machinery Reg, Blue Guide, ENISA, NIST, OECD).
// Update chunk counts after running ingest-industry-compliance.sh.
const COLLECTION_TOTALS = {
bp_compliance_gesetze: 58304,
bp_compliance_ce: 18183,
bp_legal_templates: 7689,
bp_compliance_datenschutz: 2448,
bp_dsfa_corpus: 7867,
bp_compliance_recht: 1425,
bp_nibis_eh: 7996,
total_legal: 76487, // gesetze + ce
total_all: 103912,
}
// License display labels
@@ -1444,6 +1743,8 @@ export default function RAGPage() {
const [autoRefresh, setAutoRefresh] = useState(true)
const [elapsedTime, setElapsedTime] = useState<string>('')
// Chunk browser state is now in ChunkBrowserQA component
// DSFA corpus state
const [dsfaSources, setDsfaSources] = useState<DsfaSource[]>([])
const [dsfaStatus, setDsfaStatus] = useState<DsfaCorpusStatus | null>(null)
@@ -1689,6 +1990,8 @@ export default function RAGPage() {
return () => clearInterval(interval)
}, [pipelineState?.started_at, pipelineState?.status])
// Chunk browser functions are now in ChunkBrowserQA component
const handleSearch = async () => {
if (!searchQuery.trim()) return
@@ -1774,6 +2077,7 @@ export default function RAGPage() {
{ id: 'regulations' as TabId, name: 'Regulierungen', icon: '📜' },
{ id: 'map' as TabId, name: 'Landkarte', icon: '🗺️' },
{ id: 'search' as TabId, name: 'Suche', icon: '🔍' },
{ id: 'chunks' as TabId, name: 'Chunk-Browser', icon: '🧩' },
{ id: 'data' as TabId, name: 'Daten', icon: '📁' },
{ id: 'ingestion' as TabId, name: 'Ingestion', icon: '⚙️' },
{ id: 'pipeline' as TabId, name: 'Pipeline', icon: '🔄' },
@@ -1804,7 +2108,7 @@ export default function RAGPage() {
{/* Page Purpose */}
<PagePurpose
title="Daten & RAG"
purpose="Verwalten und durchsuchen Sie 4 RAG-Collections: Legal Corpus (24 Regulierungen), DSFA Corpus (70+ Quellen inkl. internationaler Datenschutzgesetze), NiBiS EH (Bildungsinhalte) und Legal Templates (Dokumentvorlagen). Teil der KI-Daten-Pipeline fuer Compliance und Klausur-Korrektur."
purpose={`Verwalten und durchsuchen Sie 7 RAG-Collections mit ${REGULATIONS.length} Regulierungen (${Object.keys(REGULATIONS_IN_RAG).length} im RAG). Legal Corpus, DSFA Corpus (70+ Quellen), NiBiS EH (Bildungsinhalte) und Legal Templates. Teil der KI-Daten-Pipeline fuer Compliance und Klausur-Korrektur.`}
audience={['DSB', 'Compliance Officer', 'Entwickler']}
gdprArticles={['§5 UrhG (Amtliche Werke)', 'Art. 5 DSGVO (Rechenschaftspflicht)']}
architecture={{
@@ -1826,8 +2130,8 @@ export default function RAGPage() {
<div className="grid grid-cols-2 md:grid-cols-4 gap-4 mb-6">
<div className="bg-white rounded-xl p-4 border border-slate-200">
<p className="text-xs font-medium text-blue-600 uppercase mb-1">Legal Corpus</p>
<p className="text-2xl font-bold text-slate-900">{loading ? '-' : getTotalChunks().toLocaleString()}</p>
<p className="text-xs text-slate-500">Chunks &middot; {REGULATIONS.length} Regulierungen</p>
<p className="text-2xl font-bold text-slate-900">{COLLECTION_TOTALS.total_legal.toLocaleString()}</p>
<p className="text-xs text-slate-500">Chunks &middot; {Object.keys(REGULATIONS_IN_RAG).length}/{REGULATIONS.length} im RAG</p>
</div>
<div className="bg-white rounded-xl p-4 border border-slate-200">
<p className="text-xs font-medium text-purple-600 uppercase mb-1">DSFA Corpus</p>
@@ -1836,12 +2140,12 @@ export default function RAGPage() {
</div>
<div className="bg-white rounded-xl p-4 border border-slate-200">
<p className="text-xs font-medium text-emerald-600 uppercase mb-1">NiBiS EH</p>
<p className="text-2xl font-bold text-slate-900">28.662</p>
<p className="text-2xl font-bold text-slate-900">7.996</p>
<p className="text-xs text-slate-500">Chunks &middot; Bildungs-Erwartungshorizonte</p>
</div>
<div className="bg-white rounded-xl p-4 border border-slate-200">
<p className="text-xs font-medium text-orange-600 uppercase mb-1">Legal Templates</p>
<p className="text-2xl font-bold text-slate-900">824</p>
<p className="text-2xl font-bold text-slate-900">7.689</p>
<p className="text-xs text-slate-500">Chunks &middot; Dokumentvorlagen</p>
</div>
</div>
@@ -1876,8 +2180,8 @@ export default function RAGPage() {
className="p-4 rounded-lg border border-blue-200 bg-blue-50 hover:bg-blue-100 transition-colors text-left"
>
<p className="text-xs font-medium text-blue-600 uppercase">Gesetze & Regulierungen</p>
<p className="text-2xl font-bold text-slate-900 mt-1">{loading ? '-' : getTotalChunks().toLocaleString()}</p>
<p className="text-xs text-slate-500 mt-1">{REGULATIONS.length} Regulierungen (EU, DE, BSI)</p>
<p className="text-2xl font-bold text-slate-900 mt-1">{COLLECTION_TOTALS.total_legal.toLocaleString()}</p>
<p className="text-xs text-slate-500 mt-1">{Object.keys(REGULATIONS_IN_RAG).length}/{REGULATIONS.length} im RAG</p>
</button>
<button
onClick={() => { setRegulationCategory('dsfa'); setActiveTab('regulations') }}
@@ -1889,12 +2193,12 @@ export default function RAGPage() {
</button>
<div className="p-4 rounded-lg border border-emerald-200 bg-emerald-50 text-left">
<p className="text-xs font-medium text-emerald-600 uppercase">NiBiS EH</p>
<p className="text-2xl font-bold text-slate-900 mt-1">28.662</p>
<p className="text-2xl font-bold text-slate-900 mt-1">7.996</p>
<p className="text-xs text-slate-500 mt-1">Chunks &middot; Bildungs-Erwartungshorizonte</p>
</div>
<div className="p-4 rounded-lg border border-orange-200 bg-orange-50 text-left">
<p className="text-xs font-medium text-orange-600 uppercase">Legal Templates</p>
<p className="text-2xl font-bold text-slate-900 mt-1">824</p>
<p className="text-2xl font-bold text-slate-900 mt-1">7.689</p>
<p className="text-xs text-slate-500 mt-1">Chunks &middot; Dokumentvorlagen (VVT, TOM, DSFA)</p>
</div>
</div>
@@ -1904,12 +2208,13 @@ export default function RAGPage() {
<div className="grid grid-cols-1 md:grid-cols-4 gap-4">
{Object.entries(TYPE_LABELS).map(([type, label]) => {
const regs = REGULATIONS.filter((r) => r.type === type)
const totalChunks = regs.reduce((sum, r) => sum + getRegulationChunks(r.code), 0)
const inRagCount = regs.filter((r) => isInRag(r.code)).length
const totalChunks = regs.reduce((sum, r) => sum + getKnownChunks(r.code), 0)
return (
<div key={type} className="bg-white rounded-xl p-4 border border-slate-200">
<div className="flex items-center gap-2 mb-2">
<span className={`px-2 py-0.5 text-xs rounded ${TYPE_COLORS[type]}`}>{label}</span>
<span className="text-slate-500 text-sm">{regs.length} Dok.</span>
<span className="text-slate-500 text-sm">{inRagCount}/{regs.length} im RAG</span>
</div>
<p className="text-xl font-bold text-slate-900">{totalChunks.toLocaleString()} Chunks</p>
</div>
@@ -1923,20 +2228,25 @@ export default function RAGPage() {
<h3 className="font-semibold text-slate-900">Top Regulierungen (nach Chunks)</h3>
</div>
<div className="divide-y">
{REGULATIONS.sort((a, b) => getRegulationChunks(b.code) - getRegulationChunks(a.code))
.slice(0, 5)
{[...REGULATIONS].sort((a, b) => getKnownChunks(b.code) - getKnownChunks(a.code))
.slice(0, 10)
.map((reg) => {
const chunks = getRegulationChunks(reg.code)
const chunks = getKnownChunks(reg.code)
return (
<div key={reg.code} className="px-4 py-3 flex items-center justify-between">
<div className="flex items-center gap-3">
{isInRag(reg.code) ? (
<span className="text-green-500 text-sm"></span>
) : (
<span className="text-red-400 text-sm"></span>
)}
<span className={`px-2 py-0.5 text-xs rounded ${TYPE_COLORS[reg.type]}`}>
{TYPE_LABELS[reg.type]}
</span>
<span className="font-medium text-slate-900">{reg.name}</span>
<span className="text-slate-500 text-sm">({reg.code})</span>
</div>
<span className="font-bold text-teal-600">{chunks.toLocaleString()} Chunks</span>
<span className={`font-bold ${chunks > 0 ? 'text-teal-600' : 'text-slate-300'}`}>{chunks > 0 ? chunks.toLocaleString() + ' Chunks' : '—'}</span>
</div>
)
})}
@@ -1995,7 +2305,13 @@ export default function RAGPage() {
{regulationCategory === 'regulations' && (
<div className="bg-white rounded-xl border border-slate-200 overflow-hidden">
<div className="px-4 py-3 border-b bg-slate-50 flex items-center justify-between">
<h3 className="font-semibold text-slate-900">Alle {REGULATIONS.length} Regulierungen</h3>
<h3 className="font-semibold text-slate-900">
Alle {REGULATIONS.length} Regulierungen
<span className="ml-2 text-sm font-normal text-slate-500">
({REGULATIONS.filter(r => isInRag(r.code)).length} im RAG,{' '}
{REGULATIONS.filter(r => !isInRag(r.code)).length} ausstehend)
</span>
</h3>
<button
onClick={fetchStatus}
className="text-sm text-teal-600 hover:text-teal-700"
@@ -2007,6 +2323,7 @@ export default function RAGPage() {
<table className="w-full">
<thead className="bg-slate-50 border-b">
<tr>
<th className="px-4 py-3 text-center text-xs font-medium text-slate-500 uppercase w-12">RAG</th>
<th className="px-4 py-3 text-left text-xs font-medium text-slate-500 uppercase">Code</th>
<th className="px-4 py-3 text-left text-xs font-medium text-slate-500 uppercase">Typ</th>
<th className="px-4 py-3 text-left text-xs font-medium text-slate-500 uppercase">Name</th>
@@ -2017,17 +2334,10 @@ export default function RAGPage() {
</thead>
<tbody className="divide-y">
{REGULATIONS.map((reg) => {
const chunks = getRegulationChunks(reg.code)
const ratio = chunks / (reg.expected * 10) // Rough estimate: 10 chunks per requirement
let statusColor = 'text-red-500'
let statusIcon = '❌'
if (ratio > 0.5) {
statusColor = 'text-green-500'
statusIcon = '✓'
} else if (ratio > 0.1) {
statusColor = 'text-yellow-500'
statusIcon = '⚠'
}
const chunks = getKnownChunks(reg.code)
const inRag = isInRag(reg.code)
let statusColor = inRag ? 'text-green-500' : 'text-red-500'
let statusIcon = inRag ? '✓' : '❌'
const isExpanded = expandedRegulation === reg.code
return (
@@ -2036,6 +2346,13 @@ export default function RAGPage() {
onClick={() => setExpandedRegulation(isExpanded ? null : reg.code)}
className="hover:bg-slate-50 cursor-pointer transition-colors"
>
<td className="px-4 py-3 text-center">
{isInRag(reg.code) ? (
<span className="inline-flex items-center justify-center w-6 h-6 bg-green-100 text-green-600 rounded-full text-xs font-bold" title="Im RAG vorhanden"></span>
) : (
<span className="inline-flex items-center justify-center w-6 h-6 bg-red-50 text-red-400 rounded-full text-xs font-bold" title="Nicht im RAG"></span>
)}
</td>
<td className="px-4 py-3 font-mono font-medium text-teal-600">
<span className="inline-flex items-center gap-2">
<span className={`transform transition-transform ${isExpanded ? 'rotate-90' : ''}`}></span>
@@ -2048,13 +2365,20 @@ export default function RAGPage() {
</span>
</td>
<td className="px-4 py-3 text-slate-900">{reg.name}</td>
<td className="px-4 py-3 text-right font-bold">{chunks.toLocaleString()}</td>
<td className="px-4 py-3 text-right font-bold">
<span className={chunks > 0 && chunks < 10 && reg.expected >= 10 ? 'text-amber-600' : ''}>
{chunks.toLocaleString()}
{chunks > 0 && chunks < 10 && reg.expected >= 10 && (
<span className="ml-1 inline-block w-4 h-4 text-[10px] leading-4 text-center bg-amber-100 text-amber-700 rounded-full" title="Verdaechtig niedrig — Ingestion pruefen"></span>
)}
</span>
</td>
<td className="px-4 py-3 text-right text-slate-500">{reg.expected}</td>
<td className={`px-4 py-3 text-center ${statusColor}`}>{statusIcon}</td>
</tr>
{isExpanded && (
<tr key={`${reg.code}-detail`} className="bg-slate-50">
<td colSpan={6} className="px-4 py-4">
<td colSpan={7} className="px-4 py-4">
<div className="bg-white rounded-lg border border-slate-200 p-4 space-y-3">
<div>
<h4 className="font-semibold text-slate-900 mb-1">{reg.fullName}</h4>
@@ -2094,16 +2418,28 @@ export default function RAGPage() {
</span>
)}
</div>
<button
onClick={(e) => {
e.stopPropagation()
setSearchQuery(reg.name)
setActiveTab('search')
}}
className="text-teal-600 hover:text-teal-700 font-medium"
>
In Chunks suchen
</button>
<div className="flex items-center gap-3">
{REGULATION_SOURCES[reg.code] && (
<a
href={REGULATION_SOURCES[reg.code]}
target="_blank"
rel="noopener noreferrer"
onClick={(e) => e.stopPropagation()}
className="text-blue-600 hover:text-blue-700 font-medium"
>
Originalquelle
</a>
)}
<button
onClick={(e) => {
e.stopPropagation()
setActiveTab('chunks')
}}
className="text-teal-600 hover:text-teal-700 font-medium"
>
In Chunks suchen
</button>
</div>
</div>
</div>
</td>
@@ -2232,7 +2568,7 @@ export default function RAGPage() {
<div className="grid grid-cols-3 gap-4 mb-4">
<div className="bg-emerald-50 rounded-lg p-4 border border-emerald-200">
<p className="text-sm text-emerald-600 font-medium">Chunks</p>
<p className="text-2xl font-bold text-slate-900">28.662</p>
<p className="text-2xl font-bold text-slate-900">7.996</p>
</div>
<div className="bg-emerald-50 rounded-lg p-4 border border-emerald-200">
<p className="text-sm text-emerald-600 font-medium">Vector Size</p>
@@ -2264,7 +2600,7 @@ export default function RAGPage() {
<div className="grid grid-cols-3 gap-4 mb-4">
<div className="bg-orange-50 rounded-lg p-4 border border-orange-200">
<p className="text-sm text-orange-600 font-medium">Chunks</p>
<p className="text-2xl font-bold text-slate-900">824</p>
<p className="text-2xl font-bold text-slate-900">7.689</p>
</div>
<div className="bg-orange-50 rounded-lg p-4 border border-orange-200">
<p className="text-sm text-orange-600 font-medium">Vector Size</p>
@@ -2332,20 +2668,28 @@ export default function RAGPage() {
</div>
</div>
<div className="grid grid-cols-1 md:grid-cols-2 lg:grid-cols-3 gap-3">
{regs.map((reg) => (
{regs.map((reg) => {
const regInRag = isInRag(reg.code)
return (
<div
key={reg.code}
className="bg-white p-3 rounded-lg border border-slate-200"
className={`bg-white p-3 rounded-lg border ${regInRag ? 'border-green-200' : 'border-slate-200'}`}
>
<div className="flex items-center gap-2 mb-1">
<span className={`px-2 py-0.5 text-xs rounded ${TYPE_COLORS[reg.type]}`}>
{reg.code}
</span>
{regInRag ? (
<span className="px-1.5 py-0.5 text-[10px] font-bold bg-green-100 text-green-600 rounded">RAG</span>
) : (
<span className="px-1.5 py-0.5 text-[10px] font-bold bg-red-50 text-red-400 rounded"></span>
)}
</div>
<div className="font-medium text-sm text-slate-900">{reg.name}</div>
<div className="text-xs text-slate-500 mt-1 line-clamp-2">{reg.description}</div>
</div>
))}
)
})}
</div>
</>
)
@@ -2372,17 +2716,22 @@ export default function RAGPage() {
<div className="flex flex-wrap gap-2">
{group.regulations.map((code) => {
const reg = REGULATIONS.find(r => r.code === code)
const codeInRag = isInRag(code)
return (
<span
key={code}
className="px-3 py-1.5 bg-slate-100 rounded-full text-sm font-medium text-slate-700 hover:bg-slate-200 cursor-pointer"
className={`px-3 py-1.5 rounded-full text-sm font-medium cursor-pointer ${
codeInRag
? 'bg-green-100 text-green-700 hover:bg-green-200'
: 'bg-slate-100 text-slate-700 hover:bg-slate-200'
}`}
onClick={() => {
setActiveTab('regulations')
setExpandedRegulation(code)
}}
title={reg?.fullName || code}
title={`${reg?.fullName || code}${codeInRag ? ' (im RAG)' : ' (nicht im RAG)'}`}
>
{code}
{codeInRag ? '✓ ' : '✗ '}{code}
</span>
)
})}
@@ -2406,9 +2755,13 @@ export default function RAGPage() {
{intersection.regulations.map((code) => (
<span
key={code}
className="px-2 py-0.5 text-xs font-medium bg-teal-100 text-teal-700 rounded"
className={`px-2 py-0.5 text-xs font-medium rounded ${
isInRag(code)
? 'bg-green-100 text-green-700'
: 'bg-red-50 text-red-500'
}`}
>
{code}
{isInRag(code) ? '✓ ' : '✗ '}{code}
</span>
))}
</div>
@@ -2443,8 +2796,15 @@ export default function RAGPage() {
<tbody className="divide-y">
{REGULATIONS.map((reg) => (
<tr key={reg.code} className="hover:bg-slate-50">
<td className="px-2 py-2 font-medium text-teal-600 sticky left-0 bg-white">
{reg.code}
<td className="px-2 py-2 font-medium sticky left-0 bg-white">
<span className="flex items-center gap-1">
{isInRag(reg.code) ? (
<span className="text-green-500 text-[10px]"></span>
) : (
<span className="text-red-300 text-[10px]"></span>
)}
<span className="text-teal-600">{reg.code}</span>
</span>
</td>
{INDUSTRIES.filter(i => i.id !== 'all').map((industry) => {
const applies = INDUSTRY_REGULATION_MAP[industry.id]?.includes(reg.code)
@@ -2531,27 +2891,33 @@ export default function RAGPage() {
</div>
</div>
{/* Integrated Regulations */}
{/* RAG Coverage Overview */}
<div className="bg-white rounded-xl border border-slate-200 p-6">
<div className="flex items-center gap-3 mb-4">
<span className="text-2xl"></span>
<div>
<h3 className="font-semibold text-slate-900">Neu integrierte Regulierungen</h3>
<p className="text-sm text-slate-500">Jetzt im RAG-System verfuegbar (Stand: Januar 2025)</p>
<h3 className="font-semibold text-slate-900">RAG-Abdeckung ({Object.keys(REGULATIONS_IN_RAG).length} von {REGULATIONS.length} Regulierungen)</h3>
<p className="text-sm text-slate-500">Stand: Februar 2026 Alle im RAG-System verfuegbaren Regulierungen</p>
</div>
</div>
<div className="grid grid-cols-2 md:grid-cols-5 gap-3">
{INTEGRATED_REGULATIONS.map((reg) => (
<div key={reg.code} className="rounded-lg border border-green-200 bg-green-50 p-3 text-center">
<span className="px-2 py-1 text-sm font-bold bg-green-100 text-green-700 rounded">
{reg.code}
</span>
<p className="text-xs text-slate-600 mt-2">{reg.name}</p>
<p className="text-xs text-green-600 mt-1">Im RAG</p>
</div>
<div className="flex flex-wrap gap-2">
{REGULATIONS.filter(r => isInRag(r.code)).map((reg) => (
<span key={reg.code} className="px-2.5 py-1 text-xs font-medium bg-green-100 text-green-700 rounded-full border border-green-200">
{reg.code}
</span>
))}
</div>
<div className="mt-4 pt-4 border-t border-slate-100">
<p className="text-xs font-medium text-slate-500 mb-2">Noch nicht im RAG:</p>
<div className="flex flex-wrap gap-2">
{REGULATIONS.filter(r => !isInRag(r.code)).map((reg) => (
<span key={reg.code} className="px-2.5 py-1 text-xs font-medium bg-red-50 text-red-400 rounded-full border border-red-100">
{reg.code}
</span>
))}
</div>
</div>
</div>
{/* Potential Future Regulations */}
@@ -2714,6 +3080,10 @@ export default function RAGPage() {
</div>
)}
{activeTab === 'chunks' && (
<ChunkBrowserQA apiProxy={API_PROXY} />
)}
{activeTab === 'data' && (
<div className="space-y-6">
{/* Upload Document */}
@@ -2899,7 +3269,7 @@ export default function RAGPage() {
<span className="flex items-center gap-2 text-teal-600">
<svg className="animate-spin h-4 w-4" fill="none" viewBox="0 0 24 24">
<circle className="opacity-25" cx="12" cy="12" r="10" stroke="currentColor" strokeWidth="4" />
<path className="opacity-75" fill="currentColor" d="M4 12a8 8 0 018-8V0C5.373 0 0 5.373 0 12h4zm2 5.291A7.962 7.962 0 014 12H0c0 3.042 1.135 5.824 3 7.938l3-2.647z" />
<path className="opacity-75" fill="currentColor" d="M4 12a8 8 0 018-8V0C5.373 0 0 5.373 0 12h4zm2 5.291A7.962 7.962 0 014 12H0c0 3.042 1.135 5.7.689 3 7.938l3-2.647z" />
</svg>
Ingestion laeuft...
</span>
@@ -2969,7 +3339,7 @@ export default function RAGPage() {
{pipelineStarting ? (
<svg className="animate-spin h-4 w-4" fill="none" viewBox="0 0 24 24">
<circle className="opacity-25" cx="12" cy="12" r="10" stroke="currentColor" strokeWidth="4" />
<path className="opacity-75" fill="currentColor" d="M4 12a8 8 0 018-8V0C5.373 0 0 5.373 0 12h4zm2 5.291A7.962 7.962 0 014 12H0c0 3.042 1.135 5.824 3 7.938l3-2.647z" />
<path className="opacity-75" fill="currentColor" d="M4 12a8 8 0 018-8V0C5.373 0 0 5.373 0 12h4zm2 5.291A7.962 7.962 0 014 12H0c0 3.042 1.135 5.7.689 3 7.938l3-2.647z" />
</svg>
) : (
<svg className="w-4 h-4" fill="none" stroke="currentColor" viewBox="0 0 24 24">
@@ -2988,7 +3358,7 @@ export default function RAGPage() {
{pipelineLoading ? (
<svg className="animate-spin h-4 w-4" fill="none" viewBox="0 0 24 24">
<circle className="opacity-25" cx="12" cy="12" r="10" stroke="currentColor" strokeWidth="4" />
<path className="opacity-75" fill="currentColor" d="M4 12a8 8 0 018-8V0C5.373 0 0 5.373 0 12h4zm2 5.291A7.962 7.962 0 014 12H0c0 3.042 1.135 5.824 3 7.938l3-2.647z" />
<path className="opacity-75" fill="currentColor" d="M4 12a8 8 0 018-8V0C5.373 0 0 5.373 0 12h4zm2 5.291A7.962 7.962 0 014 12H0c0 3.042 1.135 5.7.689 3 7.938l3-2.647z" />
</svg>
) : (
<svg className="w-4 h-4" fill="none" stroke="currentColor" viewBox="0 0 24 24">
@@ -3021,7 +3391,7 @@ export default function RAGPage() {
<>
<svg className="animate-spin h-5 w-5" fill="none" viewBox="0 0 24 24">
<circle className="opacity-25" cx="12" cy="12" r="10" stroke="currentColor" strokeWidth="4" />
<path className="opacity-75" fill="currentColor" d="M4 12a8 8 0 018-8V0C5.373 0 0 5.373 0 12h4zm2 5.291A7.962 7.962 0 014 12H0c0 3.042 1.135 5.824 3 7.938l3-2.647z" />
<path className="opacity-75" fill="currentColor" d="M4 12a8 8 0 018-8V0C5.373 0 0 5.373 0 12h4zm2 5.291A7.962 7.962 0 014 12H0c0 3.042 1.135 5.7.689 3 7.938l3-2.647z" />
</svg>
Startet...
</>
@@ -3058,7 +3428,7 @@ export default function RAGPage() {
{pipelineState.status === 'running' && (
<svg className="w-6 h-6 text-blue-600 animate-spin" fill="none" viewBox="0 0 24 24">
<circle className="opacity-25" cx="12" cy="12" r="10" stroke="currentColor" strokeWidth="4" />
<path className="opacity-75" fill="currentColor" d="M4 12a8 8 0 018-8V0C5.373 0 0 5.373 0 12h4zm2 5.291A7.962 7.962 0 014 12H0c0 3.042 1.135 5.824 3 7.938l3-2.647z" />
<path className="opacity-75" fill="currentColor" d="M4 12a8 8 0 018-8V0C5.373 0 0 5.373 0 12h4zm2 5.291A7.962 7.962 0 014 12H0c0 3.042 1.135 5.7.689 3 7.938l3-2.647z" />
</svg>
)}
{pipelineState.status === 'failed' && (

View File

@@ -0,0 +1,241 @@
/**
* Shared RAG constants used by both page.tsx and ChunkBrowserQA.
* REGULATIONS_IN_RAG maps regulation codes to their Qdrant collection, chunk count, and qdrant_id.
* The qdrant_id is the actual `regulation_id` value stored in Qdrant payloads.
* REGULATION_INFO provides minimal metadata (code, name, type) for all regulations.
*/
export interface RagRegulationEntry {
collection: string
chunks: number
qdrant_id: string // The actual regulation_id value in Qdrant payload
}
export const REGULATIONS_IN_RAG: Record<string, RagRegulationEntry> = {
// === EU Verordnungen/Richtlinien (bp_compliance_ce) ===
GDPR: { collection: 'bp_compliance_ce', chunks: 423, qdrant_id: 'eu_2016_679' },
EPRIVACY: { collection: 'bp_compliance_ce', chunks: 134, qdrant_id: 'eu_2002_58' },
SCC: { collection: 'bp_compliance_ce', chunks: 330, qdrant_id: 'eu_2021_914' },
SCC_FULL_TEXT: { collection: 'bp_compliance_ce', chunks: 330, qdrant_id: 'eu_2021_914' },
AIACT: { collection: 'bp_compliance_ce', chunks: 726, qdrant_id: 'eu_2024_1689' },
CRA: { collection: 'bp_compliance_ce', chunks: 429, qdrant_id: 'eu_2024_2847' },
NIS2: { collection: 'bp_compliance_ce', chunks: 342, qdrant_id: 'eu_2022_2555' },
DGA: { collection: 'bp_compliance_ce', chunks: 508, qdrant_id: 'eu_2022_868' },
DSA: { collection: 'bp_compliance_ce', chunks: 1106, qdrant_id: 'eu_2022_2065' },
PLD: { collection: 'bp_compliance_ce', chunks: 44, qdrant_id: 'eu_1985_374' },
E_COMMERCE_RL: { collection: 'bp_compliance_ce', chunks: 197, qdrant_id: 'eu_2000_31' },
VERBRAUCHERRECHTE_RL: { collection: 'bp_compliance_ce', chunks: 266, qdrant_id: 'eu_2011_83' },
DIGITALE_INHALTE_RL: { collection: 'bp_compliance_ce', chunks: 321, qdrant_id: 'eu_2019_770' },
DMA: { collection: 'bp_compliance_ce', chunks: 701, qdrant_id: 'eu_2022_1925' },
DPF: { collection: 'bp_compliance_ce', chunks: 2464, qdrant_id: 'dpf' },
EUCSA: { collection: 'bp_compliance_ce', chunks: 558, qdrant_id: 'eucsa' },
DATAACT: { collection: 'bp_compliance_ce', chunks: 809, qdrant_id: 'dataact' },
DORA: { collection: 'bp_compliance_ce', chunks: 823, qdrant_id: 'dora' },
PSD2: { collection: 'bp_compliance_ce', chunks: 796, qdrant_id: 'psd2' },
AMLR: { collection: 'bp_compliance_ce', chunks: 1182, qdrant_id: 'amlr' },
MiCA: { collection: 'bp_compliance_ce', chunks: 1640, qdrant_id: 'mica' },
EHDS: { collection: 'bp_compliance_ce', chunks: 1212, qdrant_id: 'ehds' },
EAA: { collection: 'bp_compliance_ce', chunks: 433, qdrant_id: 'eaa' },
DSM: { collection: 'bp_compliance_ce', chunks: 416, qdrant_id: 'dsm' },
GPSR: { collection: 'bp_compliance_ce', chunks: 509, qdrant_id: 'gpsr' },
MACHINERY_REG: { collection: 'bp_compliance_ce', chunks: 1271, qdrant_id: 'eu_2023_1230' },
BLUE_GUIDE: { collection: 'bp_compliance_ce', chunks: 2271, qdrant_id: 'eu_blue_guide_2022' },
EU_IFRS_DE: { collection: 'bp_compliance_ce', chunks: 34388, qdrant_id: 'eu_2023_1803' },
EU_IFRS_EN: { collection: 'bp_compliance_ce', chunks: 34388, qdrant_id: 'eu_2023_1803' },
// International standards in bp_compliance_ce
NIST_SSDF: { collection: 'bp_compliance_ce', chunks: 111, qdrant_id: 'nist_sp_800_218' },
NIST_CSF_2: { collection: 'bp_compliance_ce', chunks: 67, qdrant_id: 'nist_csf_2_0' },
OECD_AI_PRINCIPLES: { collection: 'bp_compliance_ce', chunks: 34, qdrant_id: 'oecd_ai_principles' },
ENISA_SECURE_BY_DESIGN: { collection: 'bp_compliance_ce', chunks: 97, qdrant_id: 'cisa_secure_by_design' },
ENISA_SUPPLY_CHAIN: { collection: 'bp_compliance_ce', chunks: 110, qdrant_id: 'enisa_supply_chain_good_practices' },
ENISA_THREAT_LANDSCAPE: { collection: 'bp_compliance_ce', chunks: 118, qdrant_id: 'enisa_threat_landscape_supply_chain' },
ENISA_ICS_SCADA: { collection: 'bp_compliance_ce', chunks: 195, qdrant_id: 'enisa_ics_scada_dependencies' },
ENISA_CYBERSECURITY_2024: { collection: 'bp_compliance_ce', chunks: 22, qdrant_id: 'enisa_cybersecurity_state_2024' },
// === DE Gesetze (bp_compliance_gesetze) ===
TDDDG: { collection: 'bp_compliance_gesetze', chunks: 5, qdrant_id: 'tdddg_25' },
TMG_KOMPLETT: { collection: 'bp_compliance_gesetze', chunks: 108, qdrant_id: 'tmg_komplett' },
BDSG_FULL: { collection: 'bp_compliance_gesetze', chunks: 1056, qdrant_id: 'bdsg_2018_komplett' },
DE_DDG: { collection: 'bp_compliance_gesetze', chunks: 40, qdrant_id: 'ddg_5' },
DE_BGB_AGB: { collection: 'bp_compliance_gesetze', chunks: 4024, qdrant_id: 'bgb_komplett' },
DE_EGBGB: { collection: 'bp_compliance_gesetze', chunks: 36, qdrant_id: 'egbgb_widerruf' },
DE_HGB_RET: { collection: 'bp_compliance_gesetze', chunks: 11363, qdrant_id: 'hgb_komplett' },
DE_AO_RET: { collection: 'bp_compliance_gesetze', chunks: 9669, qdrant_id: 'ao_komplett' },
DE_TKG: { collection: 'bp_compliance_gesetze', chunks: 1631, qdrant_id: 'de_tkg' },
DE_DLINFOV: { collection: 'bp_compliance_gesetze', chunks: 21, qdrant_id: 'de_dlinfov' },
DE_BETRVG: { collection: 'bp_compliance_gesetze', chunks: 498, qdrant_id: 'de_betrvg' },
DE_GESCHGEHG: { collection: 'bp_compliance_gesetze', chunks: 63, qdrant_id: 'de_geschgehg' },
DE_USTG_RET: { collection: 'bp_compliance_gesetze', chunks: 1071, qdrant_id: 'de_ustg_ret' },
DE_URHG: { collection: 'bp_compliance_gesetze', chunks: 626, qdrant_id: 'urhg_komplett' },
// === BSI Standards (bp_compliance_gesetze) ===
'BSI-TR-03161-1': { collection: 'bp_compliance_gesetze', chunks: 138, qdrant_id: 'bsi_tr_03161_1' },
'BSI-TR-03161-2': { collection: 'bp_compliance_gesetze', chunks: 124, qdrant_id: 'bsi_tr_03161_2' },
'BSI-TR-03161-3': { collection: 'bp_compliance_gesetze', chunks: 121, qdrant_id: 'bsi_tr_03161_3' },
// === AT Gesetze (bp_compliance_gesetze) ===
AT_DSG: { collection: 'bp_compliance_gesetze', chunks: 805, qdrant_id: 'at_dsg' },
AT_DSG_FULL: { collection: 'bp_compliance_gesetze', chunks: 6, qdrant_id: 'at_dsg_full' },
AT_ECG: { collection: 'bp_compliance_gesetze', chunks: 120, qdrant_id: 'at_ecg' },
AT_TKG: { collection: 'bp_compliance_gesetze', chunks: 4348, qdrant_id: 'at_tkg' },
AT_KSCHG: { collection: 'bp_compliance_gesetze', chunks: 402, qdrant_id: 'at_kschg' },
AT_FAGG: { collection: 'bp_compliance_gesetze', chunks: 2, qdrant_id: 'at_fagg' },
AT_UGB_RET: { collection: 'bp_compliance_gesetze', chunks: 2828, qdrant_id: 'at_ugb_ret' },
AT_BAO_RET: { collection: 'bp_compliance_gesetze', chunks: 2246, qdrant_id: 'at_bao_ret' },
AT_MEDIENG: { collection: 'bp_compliance_gesetze', chunks: 571, qdrant_id: 'at_medieng' },
AT_ABGB_AGB: { collection: 'bp_compliance_gesetze', chunks: 2521, qdrant_id: 'at_abgb_agb' },
AT_UWG: { collection: 'bp_compliance_gesetze', chunks: 403, qdrant_id: 'at_uwg' },
// === CH Gesetze (bp_compliance_gesetze) ===
CH_DSG: { collection: 'bp_compliance_gesetze', chunks: 180, qdrant_id: 'ch_revdsg' },
CH_DSV: { collection: 'bp_compliance_gesetze', chunks: 5, qdrant_id: 'ch_dsv' },
CH_OR_AGB: { collection: 'bp_compliance_gesetze', chunks: 5, qdrant_id: 'ch_or_agb' },
CH_GEBUV: { collection: 'bp_compliance_gesetze', chunks: 5, qdrant_id: 'ch_gebuv' },
CH_ZERTES: { collection: 'bp_compliance_gesetze', chunks: 5, qdrant_id: 'ch_zertes' },
CH_ZGB_PERS: { collection: 'bp_compliance_gesetze', chunks: 5, qdrant_id: 'ch_zgb_pers' },
// === Nationale Gesetze (andere EU) in bp_compliance_gesetze ===
ES_LOPDGDD: { collection: 'bp_compliance_gesetze', chunks: 782, qdrant_id: 'es_lopdgdd' },
IT_CODICE_PRIVACY: { collection: 'bp_compliance_gesetze', chunks: 59, qdrant_id: 'it_codice_privacy' },
NL_UAVG: { collection: 'bp_compliance_gesetze', chunks: 523, qdrant_id: 'nl_uavg' },
FR_CNIL_GUIDE: { collection: 'bp_compliance_gesetze', chunks: 562, qdrant_id: 'fr_loi_informatique' },
IE_DPA_2018: { collection: 'bp_compliance_gesetze', chunks: 64, qdrant_id: 'ie_dpa_2018' },
UK_DPA_2018: { collection: 'bp_compliance_gesetze', chunks: 156, qdrant_id: 'uk_dpa_2018' },
UK_GDPR: { collection: 'bp_compliance_gesetze', chunks: 45, qdrant_id: 'uk_gdpr' },
NO_PERSONOPPLYSNINGSLOVEN: { collection: 'bp_compliance_gesetze', chunks: 41, qdrant_id: 'no_pol' },
SE_DATASKYDDSLAG: { collection: 'bp_compliance_gesetze', chunks: 56, qdrant_id: 'se_dataskyddslag' },
PL_UODO: { collection: 'bp_compliance_gesetze', chunks: 39, qdrant_id: 'pl_ustawa' },
CZ_ZOU: { collection: 'bp_compliance_gesetze', chunks: 238, qdrant_id: 'cz_zakon' },
HU_INFOTV: { collection: 'bp_compliance_gesetze', chunks: 747, qdrant_id: 'hu_info_tv' },
LU_DPA_LAW: { collection: 'bp_compliance_gesetze', chunks: 2, qdrant_id: 'lu_dpa_law' },
// === EDPB Guidelines (bp_compliance_datenschutz) ===
EDPB_GUIDELINES_5_2020: { collection: 'bp_compliance_datenschutz', chunks: 236, qdrant_id: 'edpb_05_2020' },
EDPB_GUIDELINES_7_2020: { collection: 'bp_compliance_datenschutz', chunks: 347, qdrant_id: 'edpb_guidelines_7_2020' },
EDPB_GUIDELINES_1_2020: { collection: 'bp_compliance_datenschutz', chunks: 337, qdrant_id: 'edpb_01_2020' },
EDPB_GUIDELINES_1_2022: { collection: 'bp_compliance_datenschutz', chunks: 510, qdrant_id: 'edpb_01_2022' },
EDPB_GUIDELINES_2_2023: { collection: 'bp_compliance_datenschutz', chunks: 94, qdrant_id: 'edpb_02_2023' },
EDPB_GUIDELINES_2_2024: { collection: 'bp_compliance_datenschutz', chunks: 79, qdrant_id: 'edpb_02_2024' },
EDPB_GUIDELINES_4_2019: { collection: 'bp_compliance_datenschutz', chunks: 202, qdrant_id: 'edpb_04_2019' },
EDPB_GUIDELINES_9_2022: { collection: 'bp_compliance_datenschutz', chunks: 243, qdrant_id: 'edpb_09_2022' },
EDPB_DPIA_LIST: { collection: 'bp_compliance_datenschutz', chunks: 29, qdrant_id: 'edpb_dpia_list' },
EDPB_LEGITIMATE_INTEREST: { collection: 'bp_compliance_datenschutz', chunks: 336, qdrant_id: 'edpb_legitimate_interest' },
EDPS_DPIA_LIST: { collection: 'bp_compliance_datenschutz', chunks: 35, qdrant_id: 'edps_dpia_list' },
}
/**
* Minimal regulation info for sidebar display.
* Full REGULATIONS array with descriptions remains in page.tsx.
*/
export interface RegulationInfo {
code: string
name: string
type: string
}
export const REGULATION_INFO: RegulationInfo[] = [
// EU Verordnungen
{ code: 'GDPR', name: 'DSGVO', type: 'eu_regulation' },
{ code: 'EPRIVACY', name: 'ePrivacy-Richtlinie', type: 'eu_directive' },
{ code: 'SCC', name: 'Standardvertragsklauseln', type: 'eu_regulation' },
{ code: 'SCC_FULL_TEXT', name: 'SCC Volltext', type: 'eu_regulation' },
{ code: 'DPF', name: 'EU-US Data Privacy Framework', type: 'eu_regulation' },
{ code: 'AIACT', name: 'EU AI Act', type: 'eu_regulation' },
{ code: 'CRA', name: 'Cyber Resilience Act', type: 'eu_regulation' },
{ code: 'NIS2', name: 'NIS2-Richtlinie', type: 'eu_directive' },
{ code: 'EUCSA', name: 'EU Cybersecurity Act', type: 'eu_regulation' },
{ code: 'DATAACT', name: 'Data Act', type: 'eu_regulation' },
{ code: 'DGA', name: 'Data Governance Act', type: 'eu_regulation' },
{ code: 'DSA', name: 'Digital Services Act', type: 'eu_regulation' },
{ code: 'DMA', name: 'Digital Markets Act', type: 'eu_regulation' },
{ code: 'EAA', name: 'European Accessibility Act', type: 'eu_directive' },
{ code: 'DSM', name: 'DSM-Urheberrechtsrichtlinie', type: 'eu_directive' },
{ code: 'PLD', name: 'Produkthaftungsrichtlinie', type: 'eu_directive' },
{ code: 'GPSR', name: 'General Product Safety', type: 'eu_regulation' },
{ code: 'E_COMMERCE_RL', name: 'E-Commerce-Richtlinie', type: 'eu_directive' },
{ code: 'VERBRAUCHERRECHTE_RL', name: 'Verbraucherrechte-RL', type: 'eu_directive' },
{ code: 'DIGITALE_INHALTE_RL', name: 'Digitale-Inhalte-RL', type: 'eu_directive' },
// Financial
{ code: 'DORA', name: 'DORA', type: 'eu_regulation' },
{ code: 'PSD2', name: 'PSD2', type: 'eu_directive' },
{ code: 'AMLR', name: 'AML-Verordnung', type: 'eu_regulation' },
{ code: 'MiCA', name: 'MiCA', type: 'eu_regulation' },
{ code: 'EHDS', name: 'EHDS', type: 'eu_regulation' },
{ code: 'MACHINERY_REG', name: 'Maschinenverordnung', type: 'eu_regulation' },
{ code: 'BLUE_GUIDE', name: 'Blue Guide', type: 'eu_regulation' },
{ code: 'EU_IFRS_DE', name: 'EU-IFRS (DE)', type: 'eu_regulation' },
{ code: 'EU_IFRS_EN', name: 'EU-IFRS (EN)', type: 'eu_regulation' },
// DE Gesetze
{ code: 'TDDDG', name: 'TDDDG', type: 'de_law' },
{ code: 'TMG_KOMPLETT', name: 'TMG', type: 'de_law' },
{ code: 'BDSG_FULL', name: 'BDSG', type: 'de_law' },
{ code: 'DE_DDG', name: 'DDG', type: 'de_law' },
{ code: 'DE_BGB_AGB', name: 'BGB/AGB', type: 'de_law' },
{ code: 'DE_EGBGB', name: 'EGBGB', type: 'de_law' },
{ code: 'DE_HGB_RET', name: 'HGB', type: 'de_law' },
{ code: 'DE_AO_RET', name: 'AO', type: 'de_law' },
{ code: 'DE_TKG', name: 'TKG', type: 'de_law' },
{ code: 'DE_DLINFOV', name: 'DL-InfoV', type: 'de_law' },
{ code: 'DE_BETRVG', name: 'BetrVG', type: 'de_law' },
{ code: 'DE_GESCHGEHG', name: 'GeschGehG', type: 'de_law' },
{ code: 'DE_USTG_RET', name: 'UStG', type: 'de_law' },
{ code: 'DE_URHG', name: 'UrhG', type: 'de_law' },
// BSI
{ code: 'BSI-TR-03161-1', name: 'BSI-TR Teil 1', type: 'bsi_standard' },
{ code: 'BSI-TR-03161-2', name: 'BSI-TR Teil 2', type: 'bsi_standard' },
{ code: 'BSI-TR-03161-3', name: 'BSI-TR Teil 3', type: 'bsi_standard' },
// AT
{ code: 'AT_DSG', name: 'DSG Oesterreich', type: 'at_law' },
{ code: 'AT_DSG_FULL', name: 'DSG Volltext', type: 'at_law' },
{ code: 'AT_ECG', name: 'ECG', type: 'at_law' },
{ code: 'AT_TKG', name: 'TKG AT', type: 'at_law' },
{ code: 'AT_KSCHG', name: 'KSchG', type: 'at_law' },
{ code: 'AT_FAGG', name: 'FAGG', type: 'at_law' },
{ code: 'AT_UGB_RET', name: 'UGB', type: 'at_law' },
{ code: 'AT_BAO_RET', name: 'BAO', type: 'at_law' },
{ code: 'AT_MEDIENG', name: 'MedienG', type: 'at_law' },
{ code: 'AT_ABGB_AGB', name: 'ABGB/AGB', type: 'at_law' },
{ code: 'AT_UWG', name: 'UWG AT', type: 'at_law' },
// CH
{ code: 'CH_DSG', name: 'DSG Schweiz', type: 'ch_law' },
{ code: 'CH_DSV', name: 'DSV', type: 'ch_law' },
{ code: 'CH_OR_AGB', name: 'OR/AGB', type: 'ch_law' },
{ code: 'CH_GEBUV', name: 'GeBuV', type: 'ch_law' },
{ code: 'CH_ZERTES', name: 'ZertES', type: 'ch_law' },
{ code: 'CH_ZGB_PERS', name: 'ZGB', type: 'ch_law' },
// Andere EU nationale
{ code: 'ES_LOPDGDD', name: 'LOPDGDD Spanien', type: 'national_law' },
{ code: 'IT_CODICE_PRIVACY', name: 'Codice Privacy Italien', type: 'national_law' },
{ code: 'NL_UAVG', name: 'UAVG Niederlande', type: 'national_law' },
{ code: 'FR_CNIL_GUIDE', name: 'CNIL Guide RGPD', type: 'national_law' },
{ code: 'IE_DPA_2018', name: 'DPA 2018 Ireland', type: 'national_law' },
{ code: 'UK_DPA_2018', name: 'DPA 2018 UK', type: 'national_law' },
{ code: 'UK_GDPR', name: 'UK GDPR', type: 'national_law' },
{ code: 'NO_PERSONOPPLYSNINGSLOVEN', name: 'Personopplysningsloven', type: 'national_law' },
{ code: 'SE_DATASKYDDSLAG', name: 'Dataskyddslag Schweden', type: 'national_law' },
{ code: 'PL_UODO', name: 'UODO Polen', type: 'national_law' },
{ code: 'CZ_ZOU', name: 'Zakon Tschechien', type: 'national_law' },
{ code: 'HU_INFOTV', name: 'Infotv. Ungarn', type: 'national_law' },
{ code: 'LU_DPA_LAW', name: 'Datenschutzgesetz Luxemburg', type: 'national_law' },
// EDPB
{ code: 'EDPB_GUIDELINES_5_2020', name: 'EDPB GL Einwilligung', type: 'eu_guideline' },
{ code: 'EDPB_GUIDELINES_7_2020', name: 'EDPB GL C/P Konzepte', type: 'eu_guideline' },
{ code: 'EDPB_GUIDELINES_1_2020', name: 'EDPB GL Fahrzeuge', type: 'eu_guideline' },
{ code: 'EDPB_GUIDELINES_1_2022', name: 'EDPB GL Bussgelder', type: 'eu_guideline' },
{ code: 'EDPB_GUIDELINES_2_2023', name: 'EDPB GL Art. 37 Scope', type: 'eu_guideline' },
{ code: 'EDPB_GUIDELINES_2_2024', name: 'EDPB GL 2024', type: 'eu_guideline' },
{ code: 'EDPB_GUIDELINES_4_2019', name: 'EDPB GL Art. 25 DPbD', type: 'eu_guideline' },
{ code: 'EDPB_GUIDELINES_9_2022', name: 'EDPB GL Datenschutzverletzung', type: 'eu_guideline' },
{ code: 'EDPB_DPIA_LIST', name: 'EDPB DPIA-Liste', type: 'eu_guideline' },
{ code: 'EDPB_LEGITIMATE_INTEREST', name: 'EDPB Berecht. Interesse', type: 'eu_guideline' },
{ code: 'EDPS_DPIA_LIST', name: 'EDPS DPIA-Liste', type: 'eu_guideline' },
// International Standards
{ code: 'NIST_SSDF', name: 'NIST SSDF', type: 'international_standard' },
{ code: 'NIST_CSF_2', name: 'NIST CSF 2.0', type: 'international_standard' },
{ code: 'OECD_AI_PRINCIPLES', name: 'OECD AI Principles', type: 'international_standard' },
{ code: 'ENISA_SECURE_BY_DESIGN', name: 'CISA Secure by Design', type: 'international_standard' },
{ code: 'ENISA_SUPPLY_CHAIN', name: 'ENISA Supply Chain', type: 'international_standard' },
{ code: 'ENISA_THREAT_LANDSCAPE', name: 'ENISA Threat Landscape', type: 'international_standard' },
{ code: 'ENISA_ICS_SCADA', name: 'ENISA ICS/SCADA', type: 'international_standard' },
{ code: 'ENISA_CYBERSECURITY_2024', name: 'ENISA Cybersecurity 2024', type: 'international_standard' },
]

View File

@@ -0,0 +1,163 @@
import { describe, it, expect, vi, beforeEach } from 'vitest'
/**
* Tests for Chunk-Browser logic:
* - Collection dropdown has all 10 collections
* - COLLECTION_TOTALS has expected keys
* - Text search highlighting logic
* - Pagination state management
*/
// Replicate the COMPLIANCE_COLLECTIONS from the dropdown
const COMPLIANCE_COLLECTIONS = [
'bp_compliance_gesetze',
'bp_compliance_ce',
'bp_compliance_datenschutz',
'bp_dsfa_corpus',
'bp_compliance_recht',
'bp_legal_templates',
'bp_compliance_gdpr',
'bp_compliance_schulrecht',
'bp_dsfa_templates',
'bp_dsfa_risks',
] as const
// Replicate COLLECTION_TOTALS from page.tsx
const COLLECTION_TOTALS: Record<string, number> = {
bp_compliance_gesetze: 58304,
bp_compliance_ce: 18183,
bp_legal_templates: 7689,
bp_compliance_datenschutz: 2448,
bp_dsfa_corpus: 7867,
bp_compliance_recht: 1425,
bp_nibis_eh: 7996,
total_legal: 76487,
total_all: 103912,
}
describe('Chunk-Browser Logic', () => {
describe('COMPLIANCE_COLLECTIONS', () => {
it('should have exactly 10 collections', () => {
expect(COMPLIANCE_COLLECTIONS).toHaveLength(10)
})
it('should include bp_compliance_ce for IFRS documents', () => {
expect(COMPLIANCE_COLLECTIONS).toContain('bp_compliance_ce')
})
it('should include bp_compliance_datenschutz for EFRAG/ENISA', () => {
expect(COMPLIANCE_COLLECTIONS).toContain('bp_compliance_datenschutz')
})
it('should include bp_compliance_gesetze as default', () => {
expect(COMPLIANCE_COLLECTIONS[0]).toBe('bp_compliance_gesetze')
})
it('should have all collection names starting with bp_', () => {
COMPLIANCE_COLLECTIONS.forEach((col) => {
expect(col).toMatch(/^bp_/)
})
})
})
describe('COLLECTION_TOTALS', () => {
it('should have bp_compliance_ce key', () => {
expect(COLLECTION_TOTALS).toHaveProperty('bp_compliance_ce')
})
it('should have bp_compliance_datenschutz key', () => {
expect(COLLECTION_TOTALS).toHaveProperty('bp_compliance_datenschutz')
})
it('should have positive counts for all collections', () => {
Object.values(COLLECTION_TOTALS).forEach((count) => {
expect(count).toBeGreaterThan(0)
})
})
it('total_all should be greater than total_legal', () => {
expect(COLLECTION_TOTALS.total_all).toBeGreaterThan(COLLECTION_TOTALS.total_legal)
})
})
describe('Text search filtering logic', () => {
const mockChunks = [
{ id: '1', text: 'DSGVO Artikel 1 Datenschutz', regulation_code: 'GDPR' },
{ id: '2', text: 'IFRS 16 Leasing Standard', regulation_code: 'EU_IFRS' },
{ id: '3', text: 'Datenschutz Grundverordnung', regulation_code: 'GDPR' },
{ id: '4', text: 'ENISA Supply Chain Security', regulation_code: 'ENISA' },
]
it('should filter chunks by text search (case insensitive)', () => {
const search = 'datenschutz'
const filtered = mockChunks.filter((c) =>
c.text.toLowerCase().includes(search.toLowerCase())
)
expect(filtered).toHaveLength(2)
})
it('should return all chunks when search is empty', () => {
const search = ''
const filtered = search
? mockChunks.filter((c) => c.text.toLowerCase().includes(search.toLowerCase()))
: mockChunks
expect(filtered).toHaveLength(4)
})
it('should return 0 chunks when no match', () => {
const search = 'blockchain'
const filtered = mockChunks.filter((c) =>
c.text.toLowerCase().includes(search.toLowerCase())
)
expect(filtered).toHaveLength(0)
})
it('should match IFRS chunks', () => {
const search = 'IFRS'
const filtered = mockChunks.filter((c) =>
c.text.toLowerCase().includes(search.toLowerCase())
)
expect(filtered).toHaveLength(1)
expect(filtered[0].regulation_code).toBe('EU_IFRS')
})
})
describe('Pagination state', () => {
it('should start at page 0', () => {
const currentPage = 0
expect(currentPage).toBe(0)
})
it('should increment page on next', () => {
let currentPage = 0
currentPage += 1
expect(currentPage).toBe(1)
})
it('should maintain offset history for back navigation', () => {
const history: (string | null)[] = []
history.push(null) // page 0 offset
history.push('uuid-20') // page 1 offset
history.push('uuid-40') // page 2 offset
// Go back to page 1
const prevOffset = history[history.length - 2]
expect(prevOffset).toBe('uuid-20')
})
it('should reset state on collection change', () => {
let chunkOffset: string | null = 'some-offset'
let chunkHistory: (string | null)[] = [null, 'uuid-1']
let chunkCurrentPage = 3
// Simulate collection change
chunkOffset = null
chunkHistory = []
chunkCurrentPage = 0
expect(chunkOffset).toBeNull()
expect(chunkHistory).toHaveLength(0)
expect(chunkCurrentPage).toBe(0)
})
})
})

View File

@@ -0,0 +1,90 @@
import { describe, it, expect } from 'vitest'
/**
* Tests for RAG page constants - REGULATIONS_IN_RAG, REGULATION_SOURCES, REGULATION_LICENSES
*
* These are defined inline in page.tsx, so we test the data structures
* by importing a subset of the expected values.
*/
// Expected IFRS entries in REGULATIONS_IN_RAG
const EXPECTED_IFRS_ENTRIES = {
EU_IFRS_DE: { collection: 'bp_compliance_ce', chunks: 0 },
EU_IFRS_EN: { collection: 'bp_compliance_ce', chunks: 0 },
EFRAG_ENDORSEMENT: { collection: 'bp_compliance_datenschutz', chunks: 0 },
}
// Expected REGULATION_SOURCES URLs
const EXPECTED_SOURCES = {
GDPR: 'https://eur-lex.europa.eu/legal-content/DE/TXT/?uri=CELEX:32016R0679',
EU_IFRS_DE: 'https://eur-lex.europa.eu/legal-content/DE/TXT/?uri=CELEX:32023R1803',
EU_IFRS_EN: 'https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32023R1803',
EFRAG_ENDORSEMENT: 'https://www.efrag.org/activities/endorsement-status-report',
ENISA_SECURE_DEV: 'https://www.enisa.europa.eu/publications/secure-development-best-practices',
NIST_SSDF: 'https://csrc.nist.gov/pubs/sp/800/218/final',
NIST_CSF: 'https://www.nist.gov/cyberframework',
OECD_AI: 'https://oecd.ai/en/ai-principles',
}
describe('RAG Page Constants', () => {
describe('IFRS entries in REGULATIONS_IN_RAG', () => {
it('should have EU_IFRS_DE entry with bp_compliance_ce collection', () => {
expect(EXPECTED_IFRS_ENTRIES.EU_IFRS_DE.collection).toBe('bp_compliance_ce')
})
it('should have EU_IFRS_EN entry with bp_compliance_ce collection', () => {
expect(EXPECTED_IFRS_ENTRIES.EU_IFRS_EN.collection).toBe('bp_compliance_ce')
})
it('should have EFRAG_ENDORSEMENT entry with bp_compliance_datenschutz collection', () => {
expect(EXPECTED_IFRS_ENTRIES.EFRAG_ENDORSEMENT.collection).toBe('bp_compliance_datenschutz')
})
})
describe('REGULATION_SOURCES URLs', () => {
it('should have valid EUR-Lex URLs for EU regulations', () => {
expect(EXPECTED_SOURCES.GDPR).toMatch(/^https:\/\/eur-lex\.europa\.eu/)
expect(EXPECTED_SOURCES.EU_IFRS_DE).toMatch(/^https:\/\/eur-lex\.europa\.eu/)
expect(EXPECTED_SOURCES.EU_IFRS_EN).toMatch(/^https:\/\/eur-lex\.europa\.eu/)
})
it('should have correct CELEX for IFRS DE (32023R1803)', () => {
expect(EXPECTED_SOURCES.EU_IFRS_DE).toContain('32023R1803')
})
it('should have correct CELEX for IFRS EN (32023R1803)', () => {
expect(EXPECTED_SOURCES.EU_IFRS_EN).toContain('32023R1803')
})
it('should have DE language for IFRS DE', () => {
expect(EXPECTED_SOURCES.EU_IFRS_DE).toContain('/DE/')
})
it('should have EN language for IFRS EN', () => {
expect(EXPECTED_SOURCES.EU_IFRS_EN).toContain('/EN/')
})
it('should have EFRAG URL for endorsement status', () => {
expect(EXPECTED_SOURCES.EFRAG_ENDORSEMENT).toMatch(/^https:\/\/www\.efrag\.org/)
})
it('should have ENISA URL for secure development', () => {
expect(EXPECTED_SOURCES.ENISA_SECURE_DEV).toMatch(/^https:\/\/www\.enisa\.europa\.eu/)
})
it('should have NIST URLs for SSDF and CSF', () => {
expect(EXPECTED_SOURCES.NIST_SSDF).toMatch(/nist\.gov/)
expect(EXPECTED_SOURCES.NIST_CSF).toMatch(/nist\.gov/)
})
it('should have OECD URL for AI principles', () => {
expect(EXPECTED_SOURCES.OECD_AI).toMatch(/oecd\.ai/)
})
it('should all be valid HTTPS URLs', () => {
Object.values(EXPECTED_SOURCES).forEach((url) => {
expect(url).toMatch(/^https:\/\//)
})
})
})
})

View File

@@ -0,0 +1,249 @@
import { describe, it, expect, vi, beforeEach } from 'vitest'
// Mock fetch globally
const mockFetch = vi.fn()
global.fetch = mockFetch
// Mock NextRequest and NextResponse
vi.mock('next/server', () => ({
NextRequest: class MockNextRequest {
url: string
constructor(url: string) {
this.url = url
}
},
NextResponse: {
json: (data: unknown, init?: { status?: number }) => ({
data,
status: init?.status || 200,
}),
},
}))
describe('Legal Corpus API Proxy', () => {
beforeEach(() => {
mockFetch.mockClear()
})
describe('scroll action', () => {
it('should call Qdrant scroll endpoint with correct collection', async () => {
const mockScrollResponse = {
result: {
points: [
{ id: 'uuid-1', payload: { text: 'DSGVO Artikel 1', regulation_code: 'GDPR' } },
{ id: 'uuid-2', payload: { text: 'DSGVO Artikel 2', regulation_code: 'GDPR' } },
],
next_page_offset: 'uuid-3',
},
}
mockFetch.mockResolvedValueOnce({
ok: true,
json: () => Promise.resolve(mockScrollResponse),
})
const { GET } = await import('../route')
const request = { url: 'http://localhost/api/legal-corpus?action=scroll&collection=bp_compliance_ce&limit=20' }
const response = await GET(request as any)
expect(mockFetch).toHaveBeenCalledTimes(1)
const calledUrl = mockFetch.mock.calls[0][0]
expect(calledUrl).toContain('/collections/bp_compliance_ce/points/scroll')
const body = JSON.parse(mockFetch.mock.calls[0][1].body)
expect(body.limit).toBe(20)
expect(body.with_payload).toBe(true)
expect(body.with_vector).toBe(false)
})
it('should pass offset parameter to Qdrant', async () => {
mockFetch.mockResolvedValueOnce({
ok: true,
json: () => Promise.resolve({ result: { points: [], next_page_offset: null } }),
})
const { GET } = await import('../route')
const request = { url: 'http://localhost/api/legal-corpus?action=scroll&collection=bp_compliance_gesetze&offset=some-uuid' }
await GET(request as any)
const body = JSON.parse(mockFetch.mock.calls[0][1].body)
expect(body.offset).toBe('some-uuid')
})
it('should limit chunks to max 100', async () => {
mockFetch.mockResolvedValueOnce({
ok: true,
json: () => Promise.resolve({ result: { points: [], next_page_offset: null } }),
})
const { GET } = await import('../route')
const request = { url: 'http://localhost/api/legal-corpus?action=scroll&collection=bp_compliance_ce&limit=500' }
await GET(request as any)
const body = JSON.parse(mockFetch.mock.calls[0][1].body)
expect(body.limit).toBe(100)
})
it('should apply text_search filter client-side', async () => {
const mockScrollResponse = {
result: {
points: [
{ id: 'uuid-1', payload: { text: 'DSGVO Artikel 1 Datenschutz' } },
{ id: 'uuid-2', payload: { text: 'IFRS Standard 16 Leasing' } },
{ id: 'uuid-3', payload: { text: 'Datenschutz Grundverordnung' } },
],
next_page_offset: null,
},
}
mockFetch.mockResolvedValueOnce({
ok: true,
json: () => Promise.resolve(mockScrollResponse),
})
const { GET } = await import('../route')
const request = { url: 'http://localhost/api/legal-corpus?action=scroll&collection=bp_compliance_ce&text_search=Datenschutz' }
const response = await GET(request as any)
// Should filter to only chunks containing "Datenschutz"
expect((response as any).data.chunks).toHaveLength(2)
expect((response as any).data.chunks[0].text).toContain('Datenschutz')
})
it('should flatten payload into chunk objects', async () => {
const mockScrollResponse = {
result: {
points: [
{
id: 'uuid-1',
payload: {
text: 'IFRS 16 Leasing',
regulation_code: 'EU_IFRS',
language: 'de',
celex: '32023R1803',
},
},
],
next_page_offset: null,
},
}
mockFetch.mockResolvedValueOnce({
ok: true,
json: () => Promise.resolve(mockScrollResponse),
})
const { GET } = await import('../route')
const request = { url: 'http://localhost/api/legal-corpus?action=scroll&collection=bp_compliance_ce' }
const response = await GET(request as any)
const chunk = (response as any).data.chunks[0]
expect(chunk.id).toBe('uuid-1')
expect(chunk.text).toBe('IFRS 16 Leasing')
expect(chunk.regulation_code).toBe('EU_IFRS')
expect(chunk.language).toBe('de')
})
it('should return next_offset from Qdrant response', async () => {
mockFetch.mockResolvedValueOnce({
ok: true,
json: () => Promise.resolve({
result: { points: [], next_page_offset: 'next-uuid' },
}),
})
const { GET } = await import('../route')
const request = { url: 'http://localhost/api/legal-corpus?action=scroll&collection=bp_compliance_ce' }
const response = await GET(request as any)
expect((response as any).data.next_offset).toBe('next-uuid')
})
it('should handle Qdrant scroll failure', async () => {
mockFetch.mockResolvedValueOnce({
ok: false,
status: 404,
})
const { GET } = await import('../route')
const request = { url: 'http://localhost/api/legal-corpus?action=scroll&collection=nonexistent' }
const response = await GET(request as any)
expect((response as any).status).toBe(404)
})
it('should apply filter when filter_key and filter_value provided', async () => {
mockFetch.mockResolvedValueOnce({
ok: true,
json: () => Promise.resolve({ result: { points: [], next_page_offset: null } }),
})
const { GET } = await import('../route')
const request = { url: 'http://localhost/api/legal-corpus?action=scroll&collection=bp_compliance_ce&filter_key=language&filter_value=de' }
await GET(request as any)
const body = JSON.parse(mockFetch.mock.calls[0][1].body)
expect(body.filter).toEqual({
must: [{ key: 'language', match: { value: 'de' } }],
})
})
it('should default collection to bp_compliance_gesetze', async () => {
mockFetch.mockResolvedValueOnce({
ok: true,
json: () => Promise.resolve({ result: { points: [], next_page_offset: null } }),
})
const { GET } = await import('../route')
const request = { url: 'http://localhost/api/legal-corpus?action=scroll' }
await GET(request as any)
const calledUrl = mockFetch.mock.calls[0][0]
expect(calledUrl).toContain('/collections/bp_compliance_gesetze/')
})
})
describe('collection-count action', () => {
it('should return points_count from Qdrant collection info', async () => {
mockFetch.mockResolvedValueOnce({
ok: true,
json: () => Promise.resolve({
result: { points_count: 55053 },
}),
})
const { GET } = await import('../route')
const request = { url: 'http://localhost/api/legal-corpus?action=collection-count&collection=bp_compliance_ce' }
const response = await GET(request as any)
expect((response as any).data.count).toBe(55053)
})
it('should return 0 when Qdrant is unavailable', async () => {
mockFetch.mockResolvedValueOnce({
ok: false,
status: 500,
})
const { GET } = await import('../route')
const request = { url: 'http://localhost/api/legal-corpus?action=collection-count&collection=bp_compliance_ce' }
const response = await GET(request as any)
expect((response as any).data.count).toBe(0)
})
it('should default to bp_compliance_gesetze collection', async () => {
mockFetch.mockResolvedValueOnce({
ok: true,
json: () => Promise.resolve({ result: { points_count: 1234 } }),
})
const { GET } = await import('../route')
const request = { url: 'http://localhost/api/legal-corpus?action=collection-count' }
await GET(request as any)
const calledUrl = mockFetch.mock.calls[0][0]
expect(calledUrl).toContain('/collections/bp_compliance_gesetze')
})
})
})

View File

@@ -66,6 +66,99 @@ export async function GET(request: NextRequest) {
url += `/traceability?chunk_id=${encodeURIComponent(chunkId || '')}&regulation=${encodeURIComponent(regulation || '')}`
break
}
case 'scroll': {
const collection = searchParams.get('collection') || 'bp_compliance_gesetze'
const limit = parseInt(searchParams.get('limit') || '20', 10)
const offsetParam = searchParams.get('offset')
const filterKey = searchParams.get('filter_key')
const filterValue = searchParams.get('filter_value')
const textSearch = searchParams.get('text_search')
const scrollBody: Record<string, unknown> = {
limit: Math.min(limit, 100),
with_payload: true,
with_vector: false,
}
if (offsetParam) {
scrollBody.offset = offsetParam
}
if (filterKey && filterValue) {
scrollBody.filter = {
must: [{ key: filterKey, match: { value: filterValue } }],
}
}
const scrollRes = await fetch(`${QDRANT_URL}/collections/${encodeURIComponent(collection)}/points/scroll`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify(scrollBody),
cache: 'no-store',
})
if (!scrollRes.ok) {
return NextResponse.json({ error: 'Qdrant scroll failed' }, { status: scrollRes.status })
}
const scrollData = await scrollRes.json()
const points = (scrollData.result?.points || []).map((p: { id: string; payload?: Record<string, unknown> }) => ({
id: p.id,
...p.payload,
}))
// Client-side text search filter
let filtered = points
if (textSearch && textSearch.trim()) {
const term = textSearch.toLowerCase()
filtered = points.filter((p: Record<string, unknown>) => {
const text = String(p.text || p.content || p.chunk_text || '')
return text.toLowerCase().includes(term)
})
}
return NextResponse.json({
chunks: filtered,
next_offset: scrollData.result?.next_page_offset || null,
total_in_page: points.length,
})
}
case 'regulation-counts-batch': {
const col = searchParams.get('collection') || 'bp_compliance_gesetze'
// Accept qdrant_ids (actual regulation_id values in Qdrant payload)
const qdrantIds = (searchParams.get('qdrant_ids') || '').split(',').filter(Boolean)
const results: Record<string, number> = {}
for (let i = 0; i < qdrantIds.length; i += 10) {
const batch = qdrantIds.slice(i, i + 10)
await Promise.all(batch.map(async (qid) => {
try {
const res = await fetch(`${QDRANT_URL}/collections/${encodeURIComponent(col)}/points/count`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
filter: { must: [{ key: 'regulation_id', match: { value: qid } }] },
exact: true,
}),
cache: 'no-store',
})
if (res.ok) {
const data = await res.json()
results[qid] = data.result?.count || 0
}
} catch { /* skip failed counts */ }
}))
}
return NextResponse.json({ counts: results })
}
case 'collection-count': {
const col = searchParams.get('collection') || 'bp_compliance_gesetze'
const countRes = await fetch(`${QDRANT_URL}/collections/${encodeURIComponent(col)}`, {
cache: 'no-store',
})
if (!countRes.ok) {
return NextResponse.json({ count: 0 })
}
const countData = await countRes.json()
return NextResponse.json({
count: countData.result?.points_count || 0,
})
}
default:
return NextResponse.json({ error: 'Unknown action' }, { status: 400 })
}

View File

@@ -1,8 +1,12 @@
import type { Metadata } from 'next'
import { Inter } from 'next/font/google'
import localFont from 'next/font/local'
import './globals.css'
const inter = Inter({ subsets: ['latin'] })
const inter = localFont({
src: '../public/fonts/Inter-VariableFont.woff2',
variable: '--font-inter',
display: 'swap',
})
export const metadata: Metadata = {
title: 'BreakPilot Admin Lehrer KI',

View File

@@ -0,0 +1,320 @@
'use client'
import { useState, useMemo } from 'react'
import type { ColumnResult, ColumnGroundTruth, PageRegion } from '@/app/(admin)/ai/ocr-pipeline/types'
interface ColumnControlsProps {
columnResult: ColumnResult | null
onRerun: () => void
onManualMode: () => void
onGtMode: () => void
onGroundTruth: (gt: ColumnGroundTruth) => void
onNext: () => void
isDetecting: boolean
savedGtColumns: PageRegion[] | null
}
const TYPE_COLORS: Record<string, string> = {
column_en: 'bg-blue-100 text-blue-700 dark:bg-blue-900/30 dark:text-blue-400',
column_de: 'bg-green-100 text-green-700 dark:bg-green-900/30 dark:text-green-400',
column_example: 'bg-orange-100 text-orange-700 dark:bg-orange-900/30 dark:text-orange-400',
column_text: 'bg-cyan-100 text-cyan-700 dark:bg-cyan-900/30 dark:text-cyan-400',
page_ref: 'bg-purple-100 text-purple-700 dark:bg-purple-900/30 dark:text-purple-400',
column_marker: 'bg-red-100 text-red-700 dark:bg-red-900/30 dark:text-red-400',
column_ignore: 'bg-gray-100 text-gray-500 dark:bg-gray-700/30 dark:text-gray-500',
header: 'bg-gray-100 text-gray-600 dark:bg-gray-700/50 dark:text-gray-400',
footer: 'bg-gray-100 text-gray-600 dark:bg-gray-700/50 dark:text-gray-400',
}
const TYPE_LABELS: Record<string, string> = {
column_en: 'EN',
column_de: 'DE',
column_example: 'Beispiel',
column_text: 'Text',
page_ref: 'Seite',
column_marker: 'Marker',
column_ignore: 'Ignorieren',
header: 'Header',
footer: 'Footer',
}
const METHOD_LABELS: Record<string, string> = {
content: 'Inhalt',
position_enhanced: 'Position',
position_fallback: 'Fallback',
}
interface DiffRow {
index: number
autoCol: PageRegion | null
gtCol: PageRegion | null
diffX: number | null
diffW: number | null
typeMismatch: boolean
}
/** Match auto columns to GT columns by overlap on X-axis (IoU > 50%) */
function computeDiff(autoCols: PageRegion[], gtCols: PageRegion[]): DiffRow[] {
const rows: DiffRow[] = []
const usedGt = new Set<number>()
const usedAuto = new Set<number>()
// Match auto → GT by best X-axis overlap
for (let ai = 0; ai < autoCols.length; ai++) {
const a = autoCols[ai]
let bestIdx = -1
let bestIoU = 0
for (let gi = 0; gi < gtCols.length; gi++) {
if (usedGt.has(gi)) continue
const g = gtCols[gi]
const overlapStart = Math.max(a.x, g.x)
const overlapEnd = Math.min(a.x + a.width, g.x + g.width)
const overlap = Math.max(0, overlapEnd - overlapStart)
const union = (a.width + g.width) - overlap
const iou = union > 0 ? overlap / union : 0
if (iou > bestIoU) {
bestIoU = iou
bestIdx = gi
}
}
if (bestIdx >= 0 && bestIoU > 0.3) {
usedGt.add(bestIdx)
usedAuto.add(ai)
const g = gtCols[bestIdx]
rows.push({
index: rows.length + 1,
autoCol: a,
gtCol: g,
diffX: g.x - a.x,
diffW: g.width - a.width,
typeMismatch: a.type !== g.type,
})
}
}
// Unmatched auto columns
for (let ai = 0; ai < autoCols.length; ai++) {
if (usedAuto.has(ai)) continue
rows.push({
index: rows.length + 1,
autoCol: autoCols[ai],
gtCol: null,
diffX: null,
diffW: null,
typeMismatch: false,
})
}
// Unmatched GT columns
for (let gi = 0; gi < gtCols.length; gi++) {
if (usedGt.has(gi)) continue
rows.push({
index: rows.length + 1,
autoCol: null,
gtCol: gtCols[gi],
diffX: null,
diffW: null,
typeMismatch: false,
})
}
return rows
}
export function ColumnControls({ columnResult, onRerun, onManualMode, onGtMode, onGroundTruth, onNext, isDetecting, savedGtColumns }: ColumnControlsProps) {
const [gtSaved, setGtSaved] = useState(false)
const diffRows = useMemo(() => {
if (!columnResult || !savedGtColumns) return null
const autoCols = columnResult.columns.filter(c => c.type.startsWith('column') || c.type === 'page_ref')
const gtCols = savedGtColumns.filter(c => c.type.startsWith('column') || c.type === 'page_ref')
return computeDiff(autoCols, gtCols)
}, [columnResult, savedGtColumns])
if (!columnResult) return null
const columns = columnResult.columns.filter((c: PageRegion) => c.type.startsWith('column') || c.type === 'page_ref')
const headerFooter = columnResult.columns.filter((c: PageRegion) => !c.type.startsWith('column') && c.type !== 'page_ref')
const handleGt = (isCorrect: boolean) => {
onGroundTruth({ is_correct: isCorrect })
setGtSaved(true)
}
return (
<div className="bg-white dark:bg-gray-800 rounded-xl border border-gray-200 dark:border-gray-700 p-4 space-y-4">
{/* Summary */}
<div className="flex items-center gap-3 flex-wrap">
<div className="text-sm text-gray-600 dark:text-gray-400">
<span className="font-medium text-gray-800 dark:text-gray-200">{columns.length} Spalten</span> erkannt
{columnResult.duration_seconds > 0 && (
<span className="ml-2 text-xs">({columnResult.duration_seconds}s)</span>
)}
</div>
<button
onClick={onRerun}
disabled={isDetecting}
className="text-xs px-2 py-1 bg-gray-100 dark:bg-gray-700 rounded hover:bg-gray-200 dark:hover:bg-gray-600 transition-colors disabled:opacity-50"
>
Erneut erkennen
</button>
<button
onClick={onManualMode}
className="text-xs px-2 py-1 bg-teal-100 text-teal-700 dark:bg-teal-900/30 dark:text-teal-400 rounded hover:bg-teal-200 dark:hover:bg-teal-900/50 transition-colors"
>
Manuell markieren
</button>
<button
onClick={onGtMode}
className="text-xs px-2 py-1 bg-amber-100 text-amber-700 dark:bg-amber-900/30 dark:text-amber-400 rounded hover:bg-amber-200 dark:hover:bg-amber-900/50 transition-colors"
>
{savedGtColumns ? 'Ground Truth bearbeiten' : 'Ground Truth eintragen'}
</button>
</div>
{/* Column list */}
<div className="space-y-2">
{columns.map((col: PageRegion, i: number) => (
<div key={i} className="flex items-center gap-3 text-sm">
<span className={`px-2 py-0.5 rounded text-xs font-medium ${TYPE_COLORS[col.type] || ''}`}>
{TYPE_LABELS[col.type] || col.type}
</span>
{col.classification_confidence != null && col.classification_confidence < 1.0 && (
<span className="text-xs font-medium text-gray-600 dark:text-gray-300">
{Math.round(col.classification_confidence * 100)}%
</span>
)}
{col.classification_method && (
<span className="text-xs text-gray-400 dark:text-gray-500">
({METHOD_LABELS[col.classification_method] || col.classification_method})
</span>
)}
<span className="text-gray-500 dark:text-gray-400 text-xs font-mono">
x={col.x} y={col.y} {col.width}x{col.height}px
</span>
</div>
))}
{headerFooter.map((r: PageRegion, i: number) => (
<div key={`hf-${i}`} className="flex items-center gap-3 text-sm">
<span className={`px-2 py-0.5 rounded text-xs font-medium ${TYPE_COLORS[r.type] || ''}`}>
{TYPE_LABELS[r.type] || r.type}
</span>
<span className="text-gray-500 dark:text-gray-400 text-xs font-mono">
x={r.x} y={r.y} {r.width}x{r.height}px
</span>
</div>
))}
</div>
{/* Diff table (Auto vs GT) */}
{diffRows && diffRows.length > 0 && (
<div className="border-t border-gray-100 dark:border-gray-700 pt-3">
<div className="text-xs font-medium text-gray-500 dark:text-gray-400 mb-2">
Vergleich: Auto vs Ground Truth
</div>
<div className="overflow-x-auto">
<table className="w-full text-xs">
<thead>
<tr className="text-gray-500 dark:text-gray-400 border-b border-gray-100 dark:border-gray-700">
<th className="text-left py-1 pr-2">#</th>
<th className="text-left py-1 pr-2">Auto (Typ, x, w)</th>
<th className="text-left py-1 pr-2">GT (Typ, x, w)</th>
<th className="text-right py-1 pr-2">Diff X</th>
<th className="text-right py-1">Diff W</th>
</tr>
</thead>
<tbody>
{diffRows.map((row) => (
<tr
key={row.index}
className={
!row.autoCol || !row.gtCol || row.typeMismatch
? 'bg-red-50 dark:bg-red-900/10'
: (row.diffX !== null && Math.abs(row.diffX) > 20) || (row.diffW !== null && Math.abs(row.diffW) > 20)
? 'bg-amber-50 dark:bg-amber-900/10'
: ''
}
>
<td className="py-1 pr-2 font-mono text-gray-400">{row.index}</td>
<td className="py-1 pr-2 font-mono">
{row.autoCol ? (
<span>
<span className={`inline-block px-1 rounded ${TYPE_COLORS[row.autoCol.type] || ''}`}>
{TYPE_LABELS[row.autoCol.type] || row.autoCol.type}
</span>
{' '}{row.autoCol.x}, {row.autoCol.width}
</span>
) : (
<span className="text-red-400">fehlt</span>
)}
</td>
<td className="py-1 pr-2 font-mono">
{row.gtCol ? (
<span>
<span className={`inline-block px-1 rounded ${TYPE_COLORS[row.gtCol.type] || ''}`}>
{TYPE_LABELS[row.gtCol.type] || row.gtCol.type}
</span>
{' '}{row.gtCol.x}, {row.gtCol.width}
</span>
) : (
<span className="text-red-400">fehlt</span>
)}
</td>
<td className="py-1 pr-2 text-right font-mono">
{row.diffX !== null ? (
<span className={Math.abs(row.diffX) > 20 ? 'text-amber-600 dark:text-amber-400' : 'text-gray-500'}>
{row.diffX > 0 ? '+' : ''}{row.diffX}
</span>
) : '—'}
</td>
<td className="py-1 text-right font-mono">
{row.diffW !== null ? (
<span className={Math.abs(row.diffW) > 20 ? 'text-amber-600 dark:text-amber-400' : 'text-gray-500'}>
{row.diffW > 0 ? '+' : ''}{row.diffW}
</span>
) : '—'}
</td>
</tr>
))}
</tbody>
</table>
</div>
</div>
)}
{/* Ground Truth + Navigation */}
<div className="flex items-center justify-between pt-2 border-t border-gray-100 dark:border-gray-700">
<div className="flex items-center gap-2">
<span className="text-sm text-gray-500 dark:text-gray-400">Spalten korrekt?</span>
{gtSaved ? (
<span className="text-xs text-green-600 dark:text-green-400">Gespeichert</span>
) : (
<>
<button
onClick={() => handleGt(true)}
className="text-xs px-3 py-1 bg-green-100 text-green-700 dark:bg-green-900/30 dark:text-green-400 rounded hover:bg-green-200 dark:hover:bg-green-900/50 transition-colors"
>
Ja
</button>
<button
onClick={() => handleGt(false)}
className="text-xs px-3 py-1 bg-red-100 text-red-700 dark:bg-red-900/30 dark:text-red-400 rounded hover:bg-red-200 dark:hover:bg-red-900/50 transition-colors"
>
Nein
</button>
</>
)}
</div>
<button
onClick={onNext}
className="px-4 py-2 bg-teal-600 text-white rounded-lg hover:bg-teal-700 transition-colors text-sm font-medium"
>
Weiter
</button>
</div>
</div>
)
}

View File

@@ -0,0 +1,209 @@
'use client'
import { useState } from 'react'
import type { DeskewResult, DeskewGroundTruth } from '@/app/(admin)/ai/ocr-pipeline/types'
interface DeskewControlsProps {
deskewResult: DeskewResult | null
showBinarized: boolean
onToggleBinarized: () => void
showGrid: boolean
onToggleGrid: () => void
onManualDeskew: (angle: number) => void
onGroundTruth: (gt: DeskewGroundTruth) => void
onNext: () => void
isApplying: boolean
}
const METHOD_LABELS: Record<string, string> = {
hough: 'Hough-Linien',
word_alignment: 'Wortausrichtung',
manual: 'Manuell',
}
export function DeskewControls({
deskewResult,
showBinarized,
onToggleBinarized,
showGrid,
onToggleGrid,
onManualDeskew,
onGroundTruth,
onNext,
isApplying,
}: DeskewControlsProps) {
const [manualAngle, setManualAngle] = useState(0)
const [gtFeedback, setGtFeedback] = useState<'correct' | 'incorrect' | null>(null)
const [gtNotes, setGtNotes] = useState('')
const [gtSaved, setGtSaved] = useState(false)
const handleGroundTruth = (isCorrect: boolean) => {
setGtFeedback(isCorrect ? 'correct' : 'incorrect')
if (isCorrect) {
onGroundTruth({ is_correct: true })
setGtSaved(true)
}
}
const handleGroundTruthIncorrect = () => {
onGroundTruth({
is_correct: false,
corrected_angle: manualAngle !== 0 ? manualAngle : undefined,
notes: gtNotes || undefined,
})
setGtSaved(true)
}
return (
<div className="space-y-4">
{/* Results */}
{deskewResult && (
<div className="bg-white dark:bg-gray-800 rounded-lg border border-gray-200 dark:border-gray-700 p-4">
<div className="flex flex-wrap items-center gap-3 text-sm">
<div>
<span className="text-gray-500">Winkel:</span>{' '}
<span className="font-mono font-medium">{deskewResult.angle_applied}°</span>
</div>
<div className="h-4 w-px bg-gray-300 dark:bg-gray-600" />
<div>
<span className="text-gray-500">Methode:</span>{' '}
<span className="inline-flex items-center px-2 py-0.5 rounded-full text-xs font-medium bg-teal-100 text-teal-700 dark:bg-teal-900/40 dark:text-teal-300">
{METHOD_LABELS[deskewResult.method_used] || deskewResult.method_used}
</span>
</div>
<div className="h-4 w-px bg-gray-300 dark:bg-gray-600" />
<div>
<span className="text-gray-500">Konfidenz:</span>{' '}
<span className="font-mono">{Math.round(deskewResult.confidence * 100)}%</span>
</div>
<div className="h-4 w-px bg-gray-300 dark:bg-gray-600" />
<div className="text-gray-400 text-xs">
Hough: {deskewResult.angle_hough}° | WA: {deskewResult.angle_word_alignment}°
</div>
</div>
{/* Toggles */}
<div className="flex gap-3 mt-3">
<button
onClick={onToggleBinarized}
className={`text-xs px-3 py-1 rounded-full border transition-colors ${
showBinarized
? 'bg-teal-100 border-teal-300 text-teal-700 dark:bg-teal-900/40 dark:border-teal-600 dark:text-teal-300'
: 'border-gray-300 text-gray-500 dark:border-gray-600 dark:text-gray-400'
}`}
>
Binarisiert anzeigen
</button>
<button
onClick={onToggleGrid}
className={`text-xs px-3 py-1 rounded-full border transition-colors ${
showGrid
? 'bg-teal-100 border-teal-300 text-teal-700 dark:bg-teal-900/40 dark:border-teal-600 dark:text-teal-300'
: 'border-gray-300 text-gray-500 dark:border-gray-600 dark:text-gray-400'
}`}
>
Raster anzeigen
</button>
</div>
</div>
)}
{/* Manual angle */}
{deskewResult && (
<div className="bg-white dark:bg-gray-800 rounded-lg border border-gray-200 dark:border-gray-700 p-4">
<div className="text-sm font-medium text-gray-700 dark:text-gray-300 mb-2">Manuelle Korrektur</div>
<div className="flex items-center gap-3">
<span className="text-xs text-gray-400 w-8 text-right">-5°</span>
<input
type="range"
min={-5}
max={5}
step={0.1}
value={manualAngle}
onChange={(e) => setManualAngle(parseFloat(e.target.value))}
className="flex-1 h-2 bg-gray-200 rounded-lg appearance-none cursor-pointer dark:bg-gray-700 accent-teal-500"
/>
<span className="text-xs text-gray-400 w-8">+5°</span>
<span className="font-mono text-sm w-14 text-right">{manualAngle.toFixed(1)}°</span>
<button
onClick={() => onManualDeskew(manualAngle)}
disabled={isApplying}
className="px-3 py-1.5 text-sm bg-teal-600 text-white rounded-md hover:bg-teal-700 disabled:opacity-50 transition-colors"
>
{isApplying ? '...' : 'Anwenden'}
</button>
</div>
</div>
)}
{/* Ground Truth */}
{deskewResult && (
<div className="bg-white dark:bg-gray-800 rounded-lg border border-gray-200 dark:border-gray-700 p-4">
<div className="text-sm font-medium text-gray-700 dark:text-gray-300 mb-2">
Rotation korrekt?
</div>
<p className="text-xs text-gray-400 mb-2">Nur die Drehung bewerten Woelbung/Verzerrung wird im naechsten Schritt korrigiert.</p>
{!gtSaved ? (
<div className="space-y-3">
<div className="flex gap-2">
<button
onClick={() => handleGroundTruth(true)}
className={`px-4 py-1.5 rounded-md text-sm font-medium transition-colors ${
gtFeedback === 'correct'
? 'bg-green-100 text-green-700 ring-2 ring-green-400'
: 'bg-gray-100 text-gray-600 hover:bg-green-50 dark:bg-gray-700 dark:text-gray-300'
}`}
>
Ja
</button>
<button
onClick={() => handleGroundTruth(false)}
className={`px-4 py-1.5 rounded-md text-sm font-medium transition-colors ${
gtFeedback === 'incorrect'
? 'bg-red-100 text-red-700 ring-2 ring-red-400'
: 'bg-gray-100 text-gray-600 hover:bg-red-50 dark:bg-gray-700 dark:text-gray-300'
}`}
>
Nein
</button>
</div>
{gtFeedback === 'incorrect' && (
<div className="space-y-2">
<textarea
value={gtNotes}
onChange={(e) => setGtNotes(e.target.value)}
placeholder="Notizen zur Korrektur..."
className="w-full text-sm border border-gray-300 dark:border-gray-600 rounded-md p-2 bg-white dark:bg-gray-900 text-gray-800 dark:text-gray-200"
rows={2}
/>
<button
onClick={handleGroundTruthIncorrect}
className="text-sm px-3 py-1 bg-red-600 text-white rounded-md hover:bg-red-700 transition-colors"
>
Feedback speichern
</button>
</div>
)}
</div>
) : (
<div className="text-sm text-green-600 dark:text-green-400">
Feedback gespeichert
</div>
)}
</div>
)}
{/* Next button */}
{deskewResult && (
<div className="flex justify-end">
<button
onClick={onNext}
className="px-6 py-2 bg-teal-600 text-white rounded-lg hover:bg-teal-700 font-medium transition-colors"
>
Uebernehmen & Weiter &rarr;
</button>
</div>
)}
</div>
)
}

View File

@@ -0,0 +1,201 @@
'use client'
import { useEffect, useState } from 'react'
import type { DewarpResult, DewarpGroundTruth } from '@/app/(admin)/ai/ocr-pipeline/types'
interface DewarpControlsProps {
dewarpResult: DewarpResult | null
showGrid: boolean
onToggleGrid: () => void
onManualDewarp: (shearDegrees: number) => void
onGroundTruth: (gt: DewarpGroundTruth) => void
onNext: () => void
isApplying: boolean
}
const METHOD_LABELS: Record<string, string> = {
vertical_edge: 'Vertikale Kanten',
manual: 'Manuell',
none: 'Keine Korrektur',
}
export function DewarpControls({
dewarpResult,
showGrid,
onToggleGrid,
onManualDewarp,
onGroundTruth,
onNext,
isApplying,
}: DewarpControlsProps) {
const [manualShear, setManualShear] = useState(0)
const [gtFeedback, setGtFeedback] = useState<'correct' | 'incorrect' | null>(null)
const [gtNotes, setGtNotes] = useState('')
const [gtSaved, setGtSaved] = useState(false)
// Initialize slider to auto-detected value when result arrives
useEffect(() => {
if (dewarpResult && dewarpResult.shear_degrees !== undefined) {
setManualShear(dewarpResult.shear_degrees)
}
}, [dewarpResult?.shear_degrees])
const handleGroundTruth = (isCorrect: boolean) => {
setGtFeedback(isCorrect ? 'correct' : 'incorrect')
if (isCorrect) {
onGroundTruth({ is_correct: true })
setGtSaved(true)
}
}
const handleGroundTruthIncorrect = () => {
onGroundTruth({
is_correct: false,
corrected_shear: manualShear !== 0 ? manualShear : undefined,
notes: gtNotes || undefined,
})
setGtSaved(true)
}
return (
<div className="space-y-4">
{/* Results */}
{dewarpResult && (
<div className="bg-white dark:bg-gray-800 rounded-lg border border-gray-200 dark:border-gray-700 p-4">
<div className="flex flex-wrap items-center gap-3 text-sm">
<div>
<span className="text-gray-500">Scherung:</span>{' '}
<span className="font-mono font-medium">{dewarpResult.shear_degrees}°</span>
</div>
<div className="h-4 w-px bg-gray-300 dark:bg-gray-600" />
<div>
<span className="text-gray-500">Methode:</span>{' '}
<span className="inline-flex items-center px-2 py-0.5 rounded-full text-xs font-medium bg-teal-100 text-teal-700 dark:bg-teal-900/40 dark:text-teal-300">
{METHOD_LABELS[dewarpResult.method_used] || dewarpResult.method_used}
</span>
</div>
<div className="h-4 w-px bg-gray-300 dark:bg-gray-600" />
<div>
<span className="text-gray-500">Konfidenz:</span>{' '}
<span className="font-mono">{Math.round(dewarpResult.confidence * 100)}%</span>
</div>
</div>
{/* Toggle */}
<div className="flex gap-3 mt-3">
<button
onClick={onToggleGrid}
className={`text-xs px-3 py-1 rounded-full border transition-colors ${
showGrid
? 'bg-teal-100 border-teal-300 text-teal-700 dark:bg-teal-900/40 dark:border-teal-600 dark:text-teal-300'
: 'border-gray-300 text-gray-500 dark:border-gray-600 dark:text-gray-400'
}`}
>
Raster anzeigen
</button>
</div>
</div>
)}
{/* Manual shear angle slider */}
{dewarpResult && (
<div className="bg-white dark:bg-gray-800 rounded-lg border border-gray-200 dark:border-gray-700 p-4">
<div className="text-sm font-medium text-gray-700 dark:text-gray-300 mb-2">Scherwinkel (manuell)</div>
<div className="flex items-center gap-3">
<span className="text-xs text-gray-400 w-10 text-right">-2.0°</span>
<input
type="range"
min={-200}
max={200}
step={5}
value={Math.round(manualShear * 100)}
onChange={(e) => setManualShear(parseInt(e.target.value) / 100)}
className="flex-1 h-2 bg-gray-200 rounded-lg appearance-none cursor-pointer dark:bg-gray-700 accent-teal-500"
/>
<span className="text-xs text-gray-400 w-10">+2.0°</span>
<span className="font-mono text-sm w-16 text-right">{manualShear.toFixed(2)}°</span>
<button
onClick={() => onManualDewarp(manualShear)}
disabled={isApplying}
className="px-3 py-1.5 text-sm bg-teal-600 text-white rounded-md hover:bg-teal-700 disabled:opacity-50 transition-colors"
>
{isApplying ? '...' : 'Anwenden'}
</button>
</div>
<p className="text-xs text-gray-400 mt-1">
Scherung der vertikalen Achse in Grad. Positiv = Spalten nach rechts kippen, negativ = nach links.
</p>
</div>
)}
{/* Ground Truth */}
{dewarpResult && (
<div className="bg-white dark:bg-gray-800 rounded-lg border border-gray-200 dark:border-gray-700 p-4">
<div className="text-sm font-medium text-gray-700 dark:text-gray-300 mb-2">
Spalten vertikal ausgerichtet?
</div>
<p className="text-xs text-gray-400 mb-2">Pruefen ob die Spaltenraender jetzt senkrecht zum Raster stehen.</p>
{!gtSaved ? (
<div className="space-y-3">
<div className="flex gap-2">
<button
onClick={() => handleGroundTruth(true)}
className={`px-4 py-1.5 rounded-md text-sm font-medium transition-colors ${
gtFeedback === 'correct'
? 'bg-green-100 text-green-700 ring-2 ring-green-400'
: 'bg-gray-100 text-gray-600 hover:bg-green-50 dark:bg-gray-700 dark:text-gray-300'
}`}
>
Ja
</button>
<button
onClick={() => handleGroundTruth(false)}
className={`px-4 py-1.5 rounded-md text-sm font-medium transition-colors ${
gtFeedback === 'incorrect'
? 'bg-red-100 text-red-700 ring-2 ring-red-400'
: 'bg-gray-100 text-gray-600 hover:bg-red-50 dark:bg-gray-700 dark:text-gray-300'
}`}
>
Nein
</button>
</div>
{gtFeedback === 'incorrect' && (
<div className="space-y-2">
<textarea
value={gtNotes}
onChange={(e) => setGtNotes(e.target.value)}
placeholder="Notizen zur Korrektur..."
className="w-full text-sm border border-gray-300 dark:border-gray-600 rounded-md p-2 bg-white dark:bg-gray-900 text-gray-800 dark:text-gray-200"
rows={2}
/>
<button
onClick={handleGroundTruthIncorrect}
className="text-sm px-3 py-1 bg-red-600 text-white rounded-md hover:bg-red-700 transition-colors"
>
Feedback speichern
</button>
</div>
)}
</div>
) : (
<div className="text-sm text-green-600 dark:text-green-400">
Feedback gespeichert
</div>
)}
</div>
)}
{/* Next button */}
{dewarpResult && (
<div className="flex justify-end">
<button
onClick={onNext}
className="px-6 py-2 bg-teal-600 text-white rounded-lg hover:bg-teal-700 font-medium transition-colors"
>
Uebernehmen & Weiter &rarr;
</button>
</div>
)}
</div>
)
}

View File

@@ -0,0 +1,143 @@
'use client'
import { useState } from 'react'
const A4_WIDTH_MM = 210
const A4_HEIGHT_MM = 297
interface ImageCompareViewProps {
originalUrl: string | null
deskewedUrl: string | null
showGrid: boolean
showGridLeft?: boolean
showBinarized: boolean
binarizedUrl: string | null
leftLabel?: string
rightLabel?: string
}
function MmGridOverlay() {
const lines: React.ReactNode[] = []
// Vertical lines every 10mm
for (let mm = 0; mm <= A4_WIDTH_MM; mm += 10) {
const x = (mm / A4_WIDTH_MM) * 100
const is50 = mm % 50 === 0
lines.push(
<line
key={`v-${mm}`}
x1={x} y1={0} x2={x} y2={100}
stroke={is50 ? 'rgba(59, 130, 246, 0.4)' : 'rgba(59, 130, 246, 0.15)'}
strokeWidth={is50 ? 0.12 : 0.05}
/>
)
// Label every 50mm
if (is50 && mm > 0) {
lines.push(
<text key={`vl-${mm}`} x={x} y={1.2} fill="rgba(59,130,246,0.6)" fontSize="1.2" textAnchor="middle">
{mm}
</text>
)
}
}
// Horizontal lines every 10mm
for (let mm = 0; mm <= A4_HEIGHT_MM; mm += 10) {
const y = (mm / A4_HEIGHT_MM) * 100
const is50 = mm % 50 === 0
lines.push(
<line
key={`h-${mm}`}
x1={0} y1={y} x2={100} y2={y}
stroke={is50 ? 'rgba(59, 130, 246, 0.4)' : 'rgba(59, 130, 246, 0.15)'}
strokeWidth={is50 ? 0.12 : 0.05}
/>
)
if (is50 && mm > 0) {
lines.push(
<text key={`hl-${mm}`} x={0.5} y={y + 0.6} fill="rgba(59,130,246,0.6)" fontSize="1.2">
{mm}
</text>
)
}
}
return (
<svg
viewBox="0 0 100 100"
preserveAspectRatio="none"
className="absolute inset-0 w-full h-full pointer-events-none"
style={{ zIndex: 10 }}
>
<g style={{ pointerEvents: 'none' }}>{lines}</g>
</svg>
)
}
export function ImageCompareView({
originalUrl,
deskewedUrl,
showGrid,
showGridLeft,
showBinarized,
binarizedUrl,
leftLabel,
rightLabel,
}: ImageCompareViewProps) {
const [leftError, setLeftError] = useState(false)
const [rightError, setRightError] = useState(false)
const rightUrl = showBinarized && binarizedUrl ? binarizedUrl : deskewedUrl
return (
<div className="grid grid-cols-1 lg:grid-cols-2 gap-4">
{/* Left: Original */}
<div className="space-y-2">
<h3 className="text-sm font-medium text-gray-500 dark:text-gray-400">{leftLabel || 'Original (unbearbeitet)'}</h3>
<div className="relative bg-gray-100 dark:bg-gray-900 rounded-lg overflow-hidden border border-gray-200 dark:border-gray-700"
style={{ aspectRatio: '210/297' }}>
{originalUrl && !leftError ? (
<>
<img
src={originalUrl}
alt="Original Scan"
className="w-full h-full object-contain"
onError={() => setLeftError(true)}
/>
{showGridLeft && <MmGridOverlay />}
</>
) : (
<div className="flex items-center justify-center h-full text-gray-400">
{leftError ? 'Fehler beim Laden' : 'Noch kein Bild'}
</div>
)}
</div>
</div>
{/* Right: Deskewed with Grid */}
<div className="space-y-2">
<h3 className="text-sm font-medium text-gray-500 dark:text-gray-400">
{rightLabel || `${showBinarized ? 'Binarisiert' : 'Begradigt'}${showGrid ? ' + Raster (mm)' : ''}`}
</h3>
<div className="relative bg-gray-100 dark:bg-gray-900 rounded-lg overflow-hidden border border-gray-200 dark:border-gray-700"
style={{ aspectRatio: '210/297' }}>
{rightUrl && !rightError ? (
<>
<img
src={rightUrl}
alt="Begradigtes Bild"
className="w-full h-full object-contain"
onError={() => setRightError(true)}
/>
{showGrid && <MmGridOverlay />}
</>
) : (
<div className="flex items-center justify-center h-full text-gray-400">
{rightError ? 'Fehler beim Laden' : 'Begradigung laeuft...'}
</div>
)}
</div>
</div>
</div>
)
}

View File

@@ -0,0 +1,359 @@
'use client'
import { useCallback, useEffect, useRef, useState } from 'react'
import type { ColumnTypeKey, PageRegion } from '@/app/(admin)/ai/ocr-pipeline/types'
const COLUMN_TYPES: { value: ColumnTypeKey; label: string }[] = [
{ value: 'column_en', label: 'EN' },
{ value: 'column_de', label: 'DE' },
{ value: 'column_example', label: 'Beispiel' },
{ value: 'column_text', label: 'Text' },
{ value: 'page_ref', label: 'Seite' },
{ value: 'column_marker', label: 'Marker' },
{ value: 'column_ignore', label: 'Ignorieren' },
]
const TYPE_OVERLAY_COLORS: Record<string, string> = {
column_en: 'rgba(59, 130, 246, 0.12)',
column_de: 'rgba(34, 197, 94, 0.12)',
column_example: 'rgba(249, 115, 22, 0.12)',
column_text: 'rgba(6, 182, 212, 0.12)',
page_ref: 'rgba(168, 85, 247, 0.12)',
column_marker: 'rgba(239, 68, 68, 0.12)',
column_ignore: 'rgba(128, 128, 128, 0.06)',
}
const TYPE_BADGE_COLORS: Record<string, string> = {
column_en: 'bg-blue-100 text-blue-700 dark:bg-blue-900/30 dark:text-blue-400',
column_de: 'bg-green-100 text-green-700 dark:bg-green-900/30 dark:text-green-400',
column_example: 'bg-orange-100 text-orange-700 dark:bg-orange-900/30 dark:text-orange-400',
column_text: 'bg-cyan-100 text-cyan-700 dark:bg-cyan-900/30 dark:text-cyan-400',
page_ref: 'bg-purple-100 text-purple-700 dark:bg-purple-900/30 dark:text-purple-400',
column_marker: 'bg-red-100 text-red-700 dark:bg-red-900/30 dark:text-red-400',
column_ignore: 'bg-gray-100 text-gray-500 dark:bg-gray-700/30 dark:text-gray-500',
}
// Default column type sequence for newly created columns
const DEFAULT_TYPE_SEQUENCE: ColumnTypeKey[] = [
'page_ref', 'column_en', 'column_de', 'column_example', 'column_text',
]
const MIN_DIVIDER_DISTANCE_PERCENT = 2 // Minimum 2% apart
interface ManualColumnEditorProps {
imageUrl: string
imageWidth: number
imageHeight: number
onApply: (columns: PageRegion[]) => void
onCancel: () => void
applying: boolean
mode?: 'manual' | 'ground-truth'
layout?: 'two-column' | 'stacked'
initialDividers?: number[]
initialColumnTypes?: ColumnTypeKey[]
}
export function ManualColumnEditor({
imageUrl,
imageWidth,
imageHeight,
onApply,
onCancel,
applying,
mode = 'manual',
layout = 'two-column',
initialDividers,
initialColumnTypes,
}: ManualColumnEditorProps) {
const containerRef = useRef<HTMLDivElement>(null)
const [dividers, setDividers] = useState<number[]>(initialDividers ?? [])
const [columnTypes, setColumnTypes] = useState<ColumnTypeKey[]>(initialColumnTypes ?? [])
const [dragging, setDragging] = useState<number | null>(null)
const [imageLoaded, setImageLoaded] = useState(false)
const isGT = mode === 'ground-truth'
// Sync columnTypes length when dividers change
useEffect(() => {
const numColumns = dividers.length + 1
setColumnTypes(prev => {
if (prev.length === numColumns) return prev
const next = [...prev]
while (next.length < numColumns) {
const idx = next.length
next.push(DEFAULT_TYPE_SEQUENCE[idx] || 'column_text')
}
while (next.length > numColumns) {
next.pop()
}
return next
})
}, [dividers.length])
const getXPercent = useCallback((clientX: number): number => {
if (!containerRef.current) return 0
const rect = containerRef.current.getBoundingClientRect()
const pct = ((clientX - rect.left) / rect.width) * 100
return Math.max(0, Math.min(100, pct))
}, [])
const canPlaceDivider = useCallback((xPct: number, excludeIndex?: number): boolean => {
for (let i = 0; i < dividers.length; i++) {
if (i === excludeIndex) continue
if (Math.abs(dividers[i] - xPct) < MIN_DIVIDER_DISTANCE_PERCENT) return false
}
return xPct > MIN_DIVIDER_DISTANCE_PERCENT && xPct < (100 - MIN_DIVIDER_DISTANCE_PERCENT)
}, [dividers])
// Click on image to add a divider
const handleImageClick = useCallback((e: React.MouseEvent) => {
if (dragging !== null) return
// Don't add if clicking on a divider handle
if ((e.target as HTMLElement).dataset.divider) return
const xPct = getXPercent(e.clientX)
if (!canPlaceDivider(xPct)) return
setDividers(prev => [...prev, xPct].sort((a, b) => a - b))
}, [dragging, getXPercent, canPlaceDivider])
// Drag handlers
const handleDividerMouseDown = useCallback((e: React.MouseEvent, index: number) => {
e.stopPropagation()
e.preventDefault()
setDragging(index)
}, [])
useEffect(() => {
if (dragging === null) return
const handleMouseMove = (e: MouseEvent) => {
const xPct = getXPercent(e.clientX)
if (canPlaceDivider(xPct, dragging)) {
setDividers(prev => {
const next = [...prev]
next[dragging] = xPct
return next.sort((a, b) => a - b)
})
}
}
const handleMouseUp = () => {
setDragging(null)
}
window.addEventListener('mousemove', handleMouseMove)
window.addEventListener('mouseup', handleMouseUp)
return () => {
window.removeEventListener('mousemove', handleMouseMove)
window.removeEventListener('mouseup', handleMouseUp)
}
}, [dragging, getXPercent, canPlaceDivider])
const removeDivider = useCallback((index: number) => {
setDividers(prev => prev.filter((_, i) => i !== index))
}, [])
const updateColumnType = useCallback((colIndex: number, type: ColumnTypeKey) => {
setColumnTypes(prev => {
const next = [...prev]
next[colIndex] = type
return next
})
}, [])
const handleApply = useCallback(() => {
// Build PageRegion array from dividers
const sorted = [...dividers].sort((a, b) => a - b)
const columns: PageRegion[] = []
for (let i = 0; i <= sorted.length; i++) {
const leftPct = i === 0 ? 0 : sorted[i - 1]
const rightPct = i === sorted.length ? 100 : sorted[i]
const x = Math.round((leftPct / 100) * imageWidth)
const w = Math.round(((rightPct - leftPct) / 100) * imageWidth)
columns.push({
type: columnTypes[i] || 'column_text',
x,
y: 0,
width: w,
height: imageHeight,
classification_confidence: 1.0,
classification_method: 'manual',
})
}
onApply(columns)
}, [dividers, columnTypes, imageWidth, imageHeight, onApply])
// Compute column regions for overlay
const sorted = [...dividers].sort((a, b) => a - b)
const columnRegions = Array.from({ length: sorted.length + 1 }, (_, i) => ({
leftPct: i === 0 ? 0 : sorted[i - 1],
rightPct: i === sorted.length ? 100 : sorted[i],
type: columnTypes[i] || 'column_text',
}))
return (
<div className="space-y-4">
{/* Layout: image + controls */}
<div className={layout === 'stacked' ? 'space-y-4' : 'grid grid-cols-2 gap-4'}>
{/* Left: Interactive image */}
<div>
<div className="flex items-center justify-between mb-1">
<div className="text-xs font-medium text-gray-500 dark:text-gray-400">
Klicken um Trennlinien zu setzen
</div>
<button
onClick={onCancel}
className="text-xs px-2 py-0.5 text-gray-500 hover:text-gray-700 dark:text-gray-400 dark:hover:text-gray-200"
>
Abbrechen
</button>
</div>
<div
ref={containerRef}
className="relative border rounded-lg overflow-hidden dark:border-gray-700 bg-gray-50 dark:bg-gray-900 cursor-crosshair select-none"
onClick={handleImageClick}
>
{/* eslint-disable-next-line @next/next/no-img-element */}
<img
src={imageUrl}
alt="Entzerrtes Bild"
className="w-full h-auto block"
draggable={false}
onLoad={() => setImageLoaded(true)}
/>
{imageLoaded && (
<>
{/* Column overlays */}
{columnRegions.map((region, i) => (
<div
key={`col-${i}`}
className="absolute top-0 bottom-0 pointer-events-none"
style={{
left: `${region.leftPct}%`,
width: `${region.rightPct - region.leftPct}%`,
backgroundColor: TYPE_OVERLAY_COLORS[region.type] || 'rgba(128,128,128,0.08)',
}}
>
<span className="absolute top-1 left-1/2 -translate-x-1/2 text-[10px] font-medium text-gray-600 dark:text-gray-300 bg-white/80 dark:bg-gray-800/80 px-1 rounded">
{i + 1}
</span>
</div>
))}
{/* Divider lines */}
{sorted.map((xPct, i) => (
<div
key={`div-${i}`}
data-divider="true"
className="absolute top-0 bottom-0 group"
style={{
left: `${xPct}%`,
transform: 'translateX(-50%)',
width: '12px',
cursor: 'col-resize',
zIndex: 10,
}}
onMouseDown={(e) => handleDividerMouseDown(e, i)}
>
{/* Visible line */}
<div
data-divider="true"
className="absolute top-0 bottom-0 left-1/2 -translate-x-1/2 w-0.5 border-l-2 border-dashed border-red-500"
/>
{/* Delete button */}
<button
data-divider="true"
onClick={(e) => {
e.stopPropagation()
removeDivider(i)
}}
className="absolute top-2 left-1/2 -translate-x-1/2 w-4 h-4 bg-red-500 text-white rounded-full text-[10px] leading-none flex items-center justify-center opacity-0 group-hover:opacity-100 transition-opacity z-20"
title="Linie entfernen"
>
x
</button>
</div>
))}
</>
)}
</div>
</div>
{/* Right: Column type assignment + actions */}
<div className="space-y-4">
<div className="text-xs font-medium text-gray-500 dark:text-gray-400 mb-1">
Spaltentypen
</div>
{dividers.length === 0 ? (
<div className="bg-white dark:bg-gray-800 rounded-xl border border-gray-200 dark:border-gray-700 p-6 text-center">
<div className="text-3xl mb-2">👆</div>
<p className="text-sm text-gray-500 dark:text-gray-400">
Klicken Sie auf das Bild links, um vertikale Trennlinien zwischen den Spalten zu setzen.
</p>
<p className="text-xs text-gray-400 dark:text-gray-500 mt-2">
Linien koennen per Drag verschoben und per Hover geloescht werden.
</p>
</div>
) : (
<div className="bg-white dark:bg-gray-800 rounded-xl border border-gray-200 dark:border-gray-700 p-4 space-y-3">
<div className="text-sm text-gray-600 dark:text-gray-400">
<span className="font-medium text-gray-800 dark:text-gray-200">
{dividers.length} Linien = {dividers.length + 1} Spalten
</span>
</div>
<div className="grid gap-2">
{columnRegions.map((region, i) => (
<div key={i} className="flex items-center gap-3">
<span className={`w-16 text-center px-2 py-0.5 rounded text-xs font-medium ${TYPE_BADGE_COLORS[region.type] || 'bg-gray-100 text-gray-600'}`}>
Spalte {i + 1}
</span>
<select
value={columnTypes[i] || 'column_text'}
onChange={(e) => updateColumnType(i, e.target.value as ColumnTypeKey)}
className="text-sm border border-gray-200 dark:border-gray-600 rounded px-2 py-1 bg-white dark:bg-gray-700 text-gray-800 dark:text-gray-200"
>
{COLUMN_TYPES.map(t => (
<option key={t.value} value={t.value}>{t.label}</option>
))}
</select>
<span className="text-xs text-gray-400 font-mono">
{Math.round(region.rightPct - region.leftPct)}%
</span>
</div>
))}
</div>
</div>
)}
{/* Action buttons */}
<div className="flex flex-col gap-2">
<button
onClick={handleApply}
disabled={dividers.length === 0 || applying}
className="w-full px-4 py-2 bg-teal-600 text-white rounded-lg hover:bg-teal-700 transition-colors text-sm font-medium disabled:opacity-50 disabled:cursor-not-allowed"
>
{applying
? 'Wird gespeichert...'
: isGT
? `${dividers.length + 1} Spalten als Ground Truth speichern`
: `${dividers.length + 1} Spalten uebernehmen`}
</button>
<button
onClick={() => setDividers([])}
disabled={dividers.length === 0}
className="text-xs px-3 py-2 text-gray-500 hover:text-gray-700 dark:text-gray-400 dark:hover:text-gray-200 disabled:opacity-50"
>
Alle Linien entfernen
</button>
</div>
</div>
</div>
</div>
)
}

View File

@@ -0,0 +1,66 @@
'use client'
import { PipelineStep } from '@/app/(admin)/ai/ocr-pipeline/types'
interface PipelineStepperProps {
steps: PipelineStep[]
currentStep: number
onStepClick: (index: number) => void
onReprocess?: (index: number) => void
}
export function PipelineStepper({ steps, currentStep, onStepClick, onReprocess }: PipelineStepperProps) {
return (
<div className="flex items-center justify-between px-4 py-3 bg-white dark:bg-gray-800 rounded-lg border border-gray-200 dark:border-gray-700">
{steps.map((step, index) => {
const isActive = index === currentStep
const isCompleted = step.status === 'completed'
const isFailed = step.status === 'failed'
const isClickable = index <= currentStep || isCompleted
return (
<div key={step.id} className="flex items-center">
{index > 0 && (
<div
className={`h-0.5 w-8 mx-1 ${
index <= currentStep ? 'bg-teal-400' : 'bg-gray-300 dark:bg-gray-600'
}`}
/>
)}
<div className="relative group">
<button
onClick={() => isClickable && onStepClick(index)}
disabled={!isClickable}
className={`flex items-center gap-1.5 px-3 py-1.5 rounded-full text-sm font-medium transition-all ${
isActive
? 'bg-teal-100 text-teal-700 dark:bg-teal-900/40 dark:text-teal-300 ring-2 ring-teal-400'
: isCompleted
? 'bg-green-100 text-green-700 dark:bg-green-900/40 dark:text-green-300'
: isFailed
? 'bg-red-100 text-red-700 dark:bg-red-900/40 dark:text-red-300'
: 'text-gray-400 dark:text-gray-500'
} ${isClickable ? 'cursor-pointer hover:opacity-80' : 'cursor-default'}`}
>
<span className="text-base">
{isCompleted ? '\u2713' : isFailed ? '\u2717' : step.icon}
</span>
<span className="hidden sm:inline">{step.name}</span>
<span className="sm:hidden">{index + 1}</span>
</button>
{/* Reprocess button — shown on completed steps on hover */}
{isCompleted && onReprocess && (
<button
onClick={(e) => { e.stopPropagation(); onReprocess(index) }}
className="absolute -top-1 -right-1 w-4 h-4 bg-orange-500 text-white rounded-full text-[9px] leading-none opacity-0 group-hover:opacity-100 transition-opacity flex items-center justify-center"
title={`Ab hier neu verarbeiten`}
>
&#x21BB;
</button>
)}
</div>
</div>
)
})}
</div>
)
}

View File

@@ -0,0 +1,341 @@
'use client'
import { useCallback, useEffect, useState } from 'react'
import type { ColumnResult, ColumnGroundTruth, PageRegion } from '@/app/(admin)/ai/ocr-pipeline/types'
import { ColumnControls } from './ColumnControls'
import { ManualColumnEditor } from './ManualColumnEditor'
import type { ColumnTypeKey } from '@/app/(admin)/ai/ocr-pipeline/types'
const KLAUSUR_API = '/klausur-api'
type ViewMode = 'normal' | 'ground-truth' | 'manual'
interface StepColumnDetectionProps {
sessionId: string | null
onNext: () => void
}
/** Convert PageRegion[] to divider percentages + column types for ManualColumnEditor */
function columnsToEditorState(
columns: PageRegion[],
imageWidth: number
): { dividers: number[]; columnTypes: ColumnTypeKey[] } {
if (!columns.length || !imageWidth) return { dividers: [], columnTypes: [] }
const sorted = [...columns].sort((a, b) => a.x - b.x)
const dividers: number[] = []
const columnTypes: ColumnTypeKey[] = sorted.map(c => c.type)
for (let i = 1; i < sorted.length; i++) {
const xPct = (sorted[i].x / imageWidth) * 100
dividers.push(xPct)
}
return { dividers, columnTypes }
}
export function StepColumnDetection({ sessionId, onNext }: StepColumnDetectionProps) {
const [columnResult, setColumnResult] = useState<ColumnResult | null>(null)
const [detecting, setDetecting] = useState(false)
const [error, setError] = useState<string | null>(null)
const [viewMode, setViewMode] = useState<ViewMode>('normal')
const [applying, setApplying] = useState(false)
const [imageDimensions, setImageDimensions] = useState<{ width: number; height: number } | null>(null)
const [savedGtColumns, setSavedGtColumns] = useState<PageRegion[] | null>(null)
// Fetch session info (image dimensions) + check for cached column result
useEffect(() => {
if (!sessionId || imageDimensions) return
const fetchSessionInfo = async () => {
try {
const infoRes = await fetch(`${KLAUSUR_API}/api/v1/ocr-pipeline/sessions/${sessionId}`)
if (infoRes.ok) {
const info = await infoRes.json()
if (info.image_width && info.image_height) {
setImageDimensions({ width: info.image_width, height: info.image_height })
}
if (info.column_result) {
setColumnResult(info.column_result)
return
}
}
} catch (e) {
console.error('Failed to fetch session info:', e)
}
// No cached result - run auto-detection
runAutoDetection()
}
fetchSessionInfo()
// eslint-disable-next-line react-hooks/exhaustive-deps
}, [sessionId])
// Load saved GT if exists
useEffect(() => {
if (!sessionId) return
const fetchGt = async () => {
try {
const res = await fetch(`${KLAUSUR_API}/api/v1/ocr-pipeline/sessions/${sessionId}/ground-truth/columns`)
if (res.ok) {
const data = await res.json()
const corrected = data.columns_gt?.corrected_columns
if (corrected) setSavedGtColumns(corrected)
}
} catch {
// No saved GT - that's fine
}
}
fetchGt()
}, [sessionId])
const runAutoDetection = useCallback(async () => {
if (!sessionId) return
setDetecting(true)
setError(null)
try {
const res = await fetch(`${KLAUSUR_API}/api/v1/ocr-pipeline/sessions/${sessionId}/columns`, {
method: 'POST',
})
if (!res.ok) {
const err = await res.json().catch(() => ({ detail: res.statusText }))
throw new Error(err.detail || 'Spaltenerkennung fehlgeschlagen')
}
const data: ColumnResult = await res.json()
setColumnResult(data)
} catch (e) {
setError(e instanceof Error ? e.message : 'Unbekannter Fehler')
} finally {
setDetecting(false)
}
}, [sessionId])
const handleRerun = useCallback(() => {
runAutoDetection()
}, [runAutoDetection])
const handleGroundTruth = useCallback(async (gt: ColumnGroundTruth) => {
if (!sessionId) return
try {
await fetch(`${KLAUSUR_API}/api/v1/ocr-pipeline/sessions/${sessionId}/ground-truth/columns`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify(gt),
})
} catch (e) {
console.error('Ground truth save failed:', e)
}
}, [sessionId])
const handleManualApply = useCallback(async (columns: PageRegion[]) => {
if (!sessionId) return
setApplying(true)
setError(null)
try {
const res = await fetch(`${KLAUSUR_API}/api/v1/ocr-pipeline/sessions/${sessionId}/columns/manual`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ columns }),
})
if (!res.ok) {
const err = await res.json().catch(() => ({ detail: res.statusText }))
throw new Error(err.detail || 'Manuelle Spalten konnten nicht gespeichert werden')
}
const data = await res.json()
setColumnResult({
columns: data.columns,
duration_seconds: data.duration_seconds ?? 0,
})
setViewMode('normal')
} catch (e) {
setError(e instanceof Error ? e.message : 'Fehler beim Speichern')
} finally {
setApplying(false)
}
}, [sessionId])
const handleGtApply = useCallback(async (columns: PageRegion[]) => {
if (!sessionId) return
setApplying(true)
setError(null)
try {
const gt: ColumnGroundTruth = {
is_correct: false,
corrected_columns: columns,
}
await fetch(`${KLAUSUR_API}/api/v1/ocr-pipeline/sessions/${sessionId}/ground-truth/columns`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify(gt),
})
setSavedGtColumns(columns)
setViewMode('normal')
} catch (e) {
setError(e instanceof Error ? e.message : 'Fehler beim Speichern')
} finally {
setApplying(false)
}
}, [sessionId])
if (!sessionId) {
return (
<div className="flex flex-col items-center justify-center py-16 text-center">
<div className="text-5xl mb-4">📊</div>
<h3 className="text-lg font-medium text-gray-700 dark:text-gray-300 mb-2">
Schritt 3: Spaltenerkennung
</h3>
<p className="text-gray-500 dark:text-gray-400 max-w-md">
Bitte zuerst Schritt 1 und 2 abschliessen.
</p>
</div>
)
}
const dewarpedUrl = `${KLAUSUR_API}/api/v1/ocr-pipeline/sessions/${sessionId}/image/dewarped`
const overlayUrl = `${KLAUSUR_API}/api/v1/ocr-pipeline/sessions/${sessionId}/image/columns-overlay`
// Pre-compute editor state from saved GT or auto columns for GT mode
const gtInitial = savedGtColumns
? columnsToEditorState(savedGtColumns, imageDimensions?.width ?? 1000)
: undefined
return (
<div className="space-y-4">
{/* Loading indicator */}
{detecting && (
<div className="flex items-center gap-2 text-teal-600 dark:text-teal-400 text-sm">
<div className="animate-spin w-4 h-4 border-2 border-teal-500 border-t-transparent rounded-full" />
Spaltenerkennung laeuft...
</div>
)}
{viewMode === 'manual' ? (
/* Manual column editor - overwrites column_result */
<ManualColumnEditor
imageUrl={dewarpedUrl}
imageWidth={imageDimensions?.width ?? 1000}
imageHeight={imageDimensions?.height ?? 1400}
onApply={handleManualApply}
onCancel={() => setViewMode('normal')}
applying={applying}
mode="manual"
/>
) : viewMode === 'ground-truth' ? (
/* GT mode: auto result (left, readonly) + GT editor (right) */
<div className="grid grid-cols-2 gap-4">
{/* Left: Auto result (readonly overlay) */}
<div>
<div className="text-xs font-medium text-gray-500 dark:text-gray-400 mb-1">
Auto-Ergebnis (readonly)
</div>
<div className="border rounded-lg overflow-hidden dark:border-gray-700 bg-gray-50 dark:bg-gray-900">
{columnResult ? (
// eslint-disable-next-line @next/next/no-img-element
<img
src={`${overlayUrl}?t=${Date.now()}`}
alt="Auto Spalten-Overlay"
className="w-full h-auto"
/>
) : (
<div className="aspect-[3/4] flex items-center justify-center text-gray-400 text-sm">
Keine Auto-Daten
</div>
)}
</div>
{/* Auto column list */}
{columnResult && (
<div className="mt-2 space-y-1">
<div className="text-xs font-medium text-gray-500 dark:text-gray-400">
Auto: {columnResult.columns.length} Spalten
</div>
{columnResult.columns
.filter(c => c.type.startsWith('column') || c.type === 'page_ref')
.map((col, i) => (
<div key={i} className="text-xs text-gray-500 dark:text-gray-400 font-mono">
{i + 1}. {col.type} x={col.x} w={col.width}
</div>
))}
</div>
)}
</div>
{/* Right: GT editor */}
<div>
<div className="text-xs font-medium text-gray-500 dark:text-gray-400 mb-1">
Ground Truth Editor
</div>
<ManualColumnEditor
imageUrl={dewarpedUrl}
imageWidth={imageDimensions?.width ?? 1000}
imageHeight={imageDimensions?.height ?? 1400}
onApply={handleGtApply}
onCancel={() => setViewMode('normal')}
applying={applying}
mode="ground-truth"
layout="stacked"
initialDividers={gtInitial?.dividers}
initialColumnTypes={gtInitial?.columnTypes}
/>
</div>
</div>
) : (
/* Normal mode: overlay (left) vs clean (right) */
<div className="grid grid-cols-2 gap-4">
<div>
<div className="text-xs font-medium text-gray-500 dark:text-gray-400 mb-1">
Mit Spalten-Overlay
</div>
<div className="border rounded-lg overflow-hidden dark:border-gray-700 bg-gray-50 dark:bg-gray-900">
{columnResult ? (
// eslint-disable-next-line @next/next/no-img-element
<img
src={`${overlayUrl}?t=${Date.now()}`}
alt="Spalten-Overlay"
className="w-full h-auto"
/>
) : (
<div className="aspect-[3/4] flex items-center justify-center text-gray-400 text-sm">
{detecting ? 'Erkenne Spalten...' : 'Keine Daten'}
</div>
)}
</div>
</div>
<div>
<div className="text-xs font-medium text-gray-500 dark:text-gray-400 mb-1">
Entzerrtes Bild
</div>
<div className="border rounded-lg overflow-hidden dark:border-gray-700 bg-gray-50 dark:bg-gray-900">
{/* eslint-disable-next-line @next/next/no-img-element */}
<img
src={dewarpedUrl}
alt="Entzerrt"
className="w-full h-auto"
/>
</div>
</div>
</div>
)}
{/* Controls */}
{viewMode === 'normal' && (
<ColumnControls
columnResult={columnResult}
onRerun={handleRerun}
onManualMode={() => setViewMode('manual')}
onGtMode={() => setViewMode('ground-truth')}
onGroundTruth={handleGroundTruth}
onNext={onNext}
isDetecting={detecting}
savedGtColumns={savedGtColumns}
/>
)}
{error && (
<div className="p-3 bg-red-50 dark:bg-red-900/20 text-red-600 dark:text-red-400 rounded-lg text-sm">
{error}
</div>
)}
</div>
)
}

View File

@@ -0,0 +1,19 @@
'use client'
export function StepCoordinates() {
return (
<div className="flex flex-col items-center justify-center py-16 text-center">
<div className="text-5xl mb-4">📍</div>
<h3 className="text-lg font-medium text-gray-700 dark:text-gray-300 mb-2">
Schritt 5: Koordinatenzuweisung
</h3>
<p className="text-gray-500 dark:text-gray-400 max-w-md">
Exakte Positionszuweisung fuer jedes Wort auf der Seite.
Dieser Schritt wird in einer zukuenftigen Version implementiert.
</p>
<div className="mt-6 px-4 py-2 bg-amber-100 dark:bg-amber-900/30 text-amber-700 dark:text-amber-400 rounded-full text-sm font-medium">
Kommt bald
</div>
</div>
)
}

View File

@@ -0,0 +1,277 @@
'use client'
import { useCallback, useEffect, useState } from 'react'
import type { DeskewGroundTruth, DeskewResult, SessionInfo } from '@/app/(admin)/ai/ocr-pipeline/types'
import { DeskewControls } from './DeskewControls'
import { ImageCompareView } from './ImageCompareView'
const KLAUSUR_API = '/klausur-api'
interface StepDeskewProps {
sessionId?: string | null
onNext: (sessionId: string) => void
}
export function StepDeskew({ sessionId: existingSessionId, onNext }: StepDeskewProps) {
const [session, setSession] = useState<SessionInfo | null>(null)
const [deskewResult, setDeskewResult] = useState<DeskewResult | null>(null)
const [uploading, setUploading] = useState(false)
const [deskewing, setDeskewing] = useState(false)
const [applying, setApplying] = useState(false)
const [showBinarized, setShowBinarized] = useState(false)
const [showGrid, setShowGrid] = useState(true)
const [error, setError] = useState<string | null>(null)
const [dragOver, setDragOver] = useState(false)
const [sessionName, setSessionName] = useState('')
// Reload session data when navigating back from a later step
useEffect(() => {
if (!existingSessionId || session) return
const loadSession = async () => {
try {
const res = await fetch(`${KLAUSUR_API}/api/v1/ocr-pipeline/sessions/${existingSessionId}`)
if (!res.ok) return
const data = await res.json()
const sessionInfo: SessionInfo = {
session_id: data.session_id,
filename: data.filename,
image_width: data.image_width,
image_height: data.image_height,
original_image_url: `${KLAUSUR_API}${data.original_image_url}`,
}
setSession(sessionInfo)
// Reconstruct deskew result from session data
if (data.deskew_result) {
const dr: DeskewResult = {
...data.deskew_result,
deskewed_image_url: `${KLAUSUR_API}/api/v1/ocr-pipeline/sessions/${existingSessionId}/image/deskewed`,
binarized_image_url: `${KLAUSUR_API}/api/v1/ocr-pipeline/sessions/${existingSessionId}/image/binarized`,
}
setDeskewResult(dr)
}
} catch (e) {
console.error('Failed to reload session:', e)
}
}
loadSession()
}, [existingSessionId, session])
const handleUpload = useCallback(async (file: File) => {
setUploading(true)
setError(null)
setDeskewResult(null)
try {
const formData = new FormData()
formData.append('file', file)
if (sessionName.trim()) {
formData.append('name', sessionName.trim())
}
const res = await fetch(`${KLAUSUR_API}/api/v1/ocr-pipeline/sessions`, {
method: 'POST',
body: formData,
})
if (!res.ok) {
const err = await res.json().catch(() => ({ detail: res.statusText }))
throw new Error(err.detail || 'Upload fehlgeschlagen')
}
const data: SessionInfo = await res.json()
// Prepend API prefix to relative URLs
data.original_image_url = `${KLAUSUR_API}${data.original_image_url}`
setSession(data)
// Auto-trigger deskew
setDeskewing(true)
const deskewRes = await fetch(`${KLAUSUR_API}/api/v1/ocr-pipeline/sessions/${data.session_id}/deskew`, {
method: 'POST',
})
if (!deskewRes.ok) {
throw new Error('Begradigung fehlgeschlagen')
}
const deskewData: DeskewResult = await deskewRes.json()
deskewData.deskewed_image_url = `${KLAUSUR_API}${deskewData.deskewed_image_url}`
deskewData.binarized_image_url = `${KLAUSUR_API}${deskewData.binarized_image_url}`
setDeskewResult(deskewData)
} catch (e) {
setError(e instanceof Error ? e.message : 'Unbekannter Fehler')
} finally {
setUploading(false)
setDeskewing(false)
}
}, [])
const handleManualDeskew = useCallback(async (angle: number) => {
if (!session) return
setApplying(true)
setError(null)
try {
const res = await fetch(`${KLAUSUR_API}/api/v1/ocr-pipeline/sessions/${session.session_id}/deskew/manual`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ angle }),
})
if (!res.ok) throw new Error('Manuelle Begradigung fehlgeschlagen')
const data = await res.json()
setDeskewResult((prev) =>
prev
? {
...prev,
angle_applied: data.angle_applied,
method_used: data.method_used,
// Force reload by appending timestamp
deskewed_image_url: `${KLAUSUR_API}${data.deskewed_image_url}?t=${Date.now()}`,
}
: null,
)
} catch (e) {
setError(e instanceof Error ? e.message : 'Fehler')
} finally {
setApplying(false)
}
}, [session])
const handleGroundTruth = useCallback(async (gt: DeskewGroundTruth) => {
if (!session) return
try {
await fetch(`${KLAUSUR_API}/api/v1/ocr-pipeline/sessions/${session.session_id}/ground-truth/deskew`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify(gt),
})
} catch (e) {
console.error('Ground truth save failed:', e)
}
}, [session])
const handleDrop = useCallback((e: React.DragEvent) => {
e.preventDefault()
setDragOver(false)
const file = e.dataTransfer.files[0]
if (file) handleUpload(file)
}, [handleUpload])
const handleFileInput = useCallback((e: React.ChangeEvent<HTMLInputElement>) => {
const file = e.target.files?.[0]
if (file) handleUpload(file)
}, [handleUpload])
// Upload area (no session yet)
if (!session) {
return (
<div className="space-y-4">
{/* Session name input */}
<div>
<label className="block text-sm font-medium text-gray-600 dark:text-gray-400 mb-1">
Session-Name (optional)
</label>
<input
type="text"
value={sessionName}
onChange={(e) => setSessionName(e.target.value)}
placeholder="z.B. Unit 3 Seite 42"
className="w-full max-w-sm px-3 py-2 text-sm border rounded-lg dark:bg-gray-800 dark:border-gray-600 dark:text-gray-200 focus:outline-none focus:ring-2 focus:ring-teal-500"
/>
</div>
<div
onDragOver={(e) => { e.preventDefault(); setDragOver(true) }}
onDragLeave={() => setDragOver(false)}
onDrop={handleDrop}
className={`border-2 border-dashed rounded-xl p-12 text-center transition-colors ${
dragOver
? 'border-teal-400 bg-teal-50 dark:bg-teal-900/20'
: 'border-gray-300 dark:border-gray-600 hover:border-teal-400'
}`}
>
{uploading ? (
<div className="text-gray-500">
<div className="animate-spin inline-block w-8 h-8 border-2 border-teal-500 border-t-transparent rounded-full mb-3" />
<p>Wird hochgeladen...</p>
</div>
) : (
<>
<div className="text-4xl mb-3">📄</div>
<p className="text-gray-600 dark:text-gray-400 mb-2">
PDF oder Bild hierher ziehen
</p>
<p className="text-sm text-gray-400 mb-4">oder</p>
<label className="inline-block px-4 py-2 bg-teal-600 text-white rounded-lg cursor-pointer hover:bg-teal-700 transition-colors">
Datei auswaehlen
<input
type="file"
accept=".pdf,.png,.jpg,.jpeg,.tiff,.tif"
onChange={handleFileInput}
className="hidden"
/>
</label>
</>
)}
</div>
{error && (
<div className="p-3 bg-red-50 dark:bg-red-900/20 text-red-600 dark:text-red-400 rounded-lg text-sm">
{error}
</div>
)}
</div>
)
}
// Session active: show comparison + controls
return (
<div className="space-y-4">
{/* Filename */}
<div className="text-sm text-gray-500 dark:text-gray-400">
Datei: <span className="font-medium text-gray-700 dark:text-gray-300">{session.filename}</span>
{' '}({session.image_width} x {session.image_height} px)
</div>
{/* Loading indicator */}
{deskewing && (
<div className="flex items-center gap-2 text-teal-600 dark:text-teal-400 text-sm">
<div className="animate-spin w-4 h-4 border-2 border-teal-500 border-t-transparent rounded-full" />
Begradigung laeuft (beide Methoden)...
</div>
)}
{/* Image comparison */}
<ImageCompareView
originalUrl={session.original_image_url}
deskewedUrl={deskewResult?.deskewed_image_url ?? null}
showGrid={showGrid}
showBinarized={showBinarized}
binarizedUrl={deskewResult?.binarized_image_url ?? null}
/>
{/* Controls */}
<DeskewControls
deskewResult={deskewResult}
showBinarized={showBinarized}
onToggleBinarized={() => setShowBinarized((v) => !v)}
showGrid={showGrid}
onToggleGrid={() => setShowGrid((v) => !v)}
onManualDeskew={handleManualDeskew}
onGroundTruth={handleGroundTruth}
onNext={() => session && onNext(session.session_id)}
isApplying={applying}
/>
{error && (
<div className="p-3 bg-red-50 dark:bg-red-900/20 text-red-600 dark:text-red-400 rounded-lg text-sm">
{error}
</div>
)}
</div>
)
}

View File

@@ -0,0 +1,151 @@
'use client'
import { useCallback, useEffect, useState } from 'react'
import type { DewarpResult, DewarpGroundTruth } from '@/app/(admin)/ai/ocr-pipeline/types'
import { DewarpControls } from './DewarpControls'
import { ImageCompareView } from './ImageCompareView'
const KLAUSUR_API = '/klausur-api'
interface StepDewarpProps {
sessionId: string | null
onNext: () => void
}
export function StepDewarp({ sessionId, onNext }: StepDewarpProps) {
const [dewarpResult, setDewarpResult] = useState<DewarpResult | null>(null)
const [dewarping, setDewarping] = useState(false)
const [applying, setApplying] = useState(false)
const [showGrid, setShowGrid] = useState(true)
const [error, setError] = useState<string | null>(null)
// Auto-trigger dewarp when component mounts with a sessionId
useEffect(() => {
if (!sessionId || dewarpResult) return
const runDewarp = async () => {
setDewarping(true)
setError(null)
try {
const res = await fetch(`${KLAUSUR_API}/api/v1/ocr-pipeline/sessions/${sessionId}/dewarp`, {
method: 'POST',
})
if (!res.ok) {
const err = await res.json().catch(() => ({ detail: res.statusText }))
throw new Error(err.detail || 'Entzerrung fehlgeschlagen')
}
const data: DewarpResult = await res.json()
data.dewarped_image_url = `${KLAUSUR_API}${data.dewarped_image_url}`
setDewarpResult(data)
} catch (e) {
setError(e instanceof Error ? e.message : 'Unbekannter Fehler')
} finally {
setDewarping(false)
}
}
runDewarp()
}, [sessionId, dewarpResult])
const handleManualDewarp = useCallback(async (shearDegrees: number) => {
if (!sessionId) return
setApplying(true)
setError(null)
try {
const res = await fetch(`${KLAUSUR_API}/api/v1/ocr-pipeline/sessions/${sessionId}/dewarp/manual`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ shear_degrees: shearDegrees }),
})
if (!res.ok) throw new Error('Manuelle Entzerrung fehlgeschlagen')
const data = await res.json()
setDewarpResult((prev) =>
prev
? {
...prev,
method_used: data.method_used,
shear_degrees: data.shear_degrees,
dewarped_image_url: `${KLAUSUR_API}${data.dewarped_image_url}?t=${Date.now()}`,
}
: null,
)
} catch (e) {
setError(e instanceof Error ? e.message : 'Fehler')
} finally {
setApplying(false)
}
}, [sessionId])
const handleGroundTruth = useCallback(async (gt: DewarpGroundTruth) => {
if (!sessionId) return
try {
await fetch(`${KLAUSUR_API}/api/v1/ocr-pipeline/sessions/${sessionId}/ground-truth/dewarp`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify(gt),
})
} catch (e) {
console.error('Ground truth save failed:', e)
}
}, [sessionId])
if (!sessionId) {
return (
<div className="flex flex-col items-center justify-center py-16 text-center">
<div className="text-5xl mb-4">🔧</div>
<h3 className="text-lg font-medium text-gray-700 dark:text-gray-300 mb-2">
Schritt 2: Entzerrung (Dewarp)
</h3>
<p className="text-gray-500 dark:text-gray-400 max-w-md">
Bitte zuerst Schritt 1 (Begradigung) abschliessen.
</p>
</div>
)
}
const deskewedUrl = `${KLAUSUR_API}/api/v1/ocr-pipeline/sessions/${sessionId}/image/deskewed`
const dewarpedUrl = dewarpResult?.dewarped_image_url ?? null
return (
<div className="space-y-4">
{/* Loading indicator */}
{dewarping && (
<div className="flex items-center gap-2 text-teal-600 dark:text-teal-400 text-sm">
<div className="animate-spin w-4 h-4 border-2 border-teal-500 border-t-transparent rounded-full" />
Entzerrung laeuft (beide Methoden)...
</div>
)}
{/* Image comparison: deskewed (left) vs dewarped (right) */}
<ImageCompareView
originalUrl={deskewedUrl}
deskewedUrl={dewarpedUrl}
showGrid={showGrid}
showGridLeft={showGrid}
showBinarized={false}
binarizedUrl={null}
leftLabel={`Begradigt (nach Deskew)${showGrid ? ' + Raster' : ''}`}
rightLabel={`Entzerrt${showGrid ? ' + Raster (mm)' : ''}`}
/>
{/* Controls */}
<DewarpControls
dewarpResult={dewarpResult}
showGrid={showGrid}
onToggleGrid={() => setShowGrid((v) => !v)}
onManualDewarp={handleManualDewarp}
onGroundTruth={handleGroundTruth}
onNext={onNext}
isApplying={applying}
/>
{error && (
<div className="p-3 bg-red-50 dark:bg-red-900/20 text-red-600 dark:text-red-400 rounded-lg text-sm">
{error}
</div>
)}
</div>
)
}

View File

@@ -0,0 +1,19 @@
'use client'
export function StepGroundTruth() {
return (
<div className="flex flex-col items-center justify-center py-16 text-center">
<div className="text-5xl mb-4"></div>
<h3 className="text-lg font-medium text-gray-700 dark:text-gray-300 mb-2">
Schritt 7: Ground Truth Validierung
</h3>
<p className="text-gray-500 dark:text-gray-400 max-w-md">
Gesamtpruefung der rekonstruierten Seite gegen das Original.
Dieser Schritt wird in einer zukuenftigen Version implementiert.
</p>
<div className="mt-6 px-4 py-2 bg-amber-100 dark:bg-amber-900/30 text-amber-700 dark:text-amber-400 rounded-full text-sm font-medium">
Kommt bald
</div>
</div>
)
}

View File

@@ -0,0 +1,656 @@
'use client'
import { useCallback, useEffect, useRef, useState } from 'react'
import type { GridResult, WordEntry } from '@/app/(admin)/ai/ocr-pipeline/types'
const KLAUSUR_API = '/klausur-api'
interface LlmChange {
row_index: number
field: 'english' | 'german' | 'example'
old: string
new: string
}
interface StepLlmReviewProps {
sessionId: string | null
onNext: () => void
}
interface ReviewMeta {
total_entries: number
to_review: number
skipped: number
model: string
skipped_indices?: number[]
}
interface StreamProgress {
current: number
total: number
}
const FIELD_LABELS: Record<string, string> = {
english: 'EN',
german: 'DE',
example: 'Beispiel',
}
type RowStatus = 'pending' | 'active' | 'reviewed' | 'corrected' | 'skipped'
export function StepLlmReview({ sessionId, onNext }: StepLlmReviewProps) {
// Core state
const [status, setStatus] = useState<'idle' | 'loading' | 'ready' | 'running' | 'done' | 'error' | 'applied'>('idle')
const [meta, setMeta] = useState<ReviewMeta | null>(null)
const [changes, setChanges] = useState<LlmChange[]>([])
const [progress, setProgress] = useState<StreamProgress | null>(null)
const [totalDuration, setTotalDuration] = useState(0)
const [error, setError] = useState('')
const [accepted, setAccepted] = useState<Set<number>>(new Set())
const [applying, setApplying] = useState(false)
// Full vocab table state
const [vocabEntries, setVocabEntries] = useState<WordEntry[]>([])
const [activeRowIndices, setActiveRowIndices] = useState<Set<number>>(new Set())
const [reviewedRows, setReviewedRows] = useState<Set<number>>(new Set())
const [skippedRows, setSkippedRows] = useState<Set<number>>(new Set())
const [correctedMap, setCorrectedMap] = useState<Map<number, LlmChange[]>>(new Map())
// Image
const [imageNaturalSize, setImageNaturalSize] = useState<{ w: number; h: number } | null>(null)
const tableRef = useRef<HTMLDivElement>(null)
const activeRowRef = useRef<HTMLTableRowElement>(null)
// Load session data on mount
useEffect(() => {
if (!sessionId) return
loadSessionData()
// eslint-disable-next-line react-hooks/exhaustive-deps
}, [sessionId])
const loadSessionData = async () => {
if (!sessionId) return
setStatus('loading')
try {
const res = await fetch(`${KLAUSUR_API}/api/v1/ocr-pipeline/sessions/${sessionId}`)
if (!res.ok) throw new Error(`HTTP ${res.status}`)
const data = await res.json()
const wordResult: GridResult | undefined = data.word_result
if (!wordResult) {
setError('Keine Worterkennungsdaten gefunden. Bitte zuerst Schritt 5 abschliessen.')
setStatus('error')
return
}
const entries = wordResult.vocab_entries || wordResult.entries || []
setVocabEntries(entries)
// Check if LLM review was already run
const llmReview = wordResult.llm_review
if (llmReview && llmReview.changes) {
const existingChanges: LlmChange[] = llmReview.changes as LlmChange[]
setChanges(existingChanges)
setTotalDuration(llmReview.duration_ms || 0)
// Mark all rows as reviewed
const allReviewed = new Set(entries.map((_: WordEntry, i: number) => i))
setReviewedRows(allReviewed)
// Build corrected map
const cMap = new Map<number, LlmChange[]>()
for (const c of existingChanges) {
const existing = cMap.get(c.row_index) || []
existing.push(c)
cMap.set(c.row_index, existing)
}
setCorrectedMap(cMap)
// Default: all accepted
setAccepted(new Set(existingChanges.map((_: LlmChange, i: number) => i)))
setMeta({
total_entries: entries.length,
to_review: llmReview.entries_corrected !== undefined ? entries.length : entries.length,
skipped: 0,
model: llmReview.model_used || 'unknown',
})
setStatus('done')
} else {
setStatus('ready')
}
} catch (e: unknown) {
setError(e instanceof Error ? e.message : String(e))
setStatus('error')
}
}
const runReview = useCallback(async () => {
if (!sessionId) return
setStatus('running')
setError('')
setChanges([])
setProgress(null)
setMeta(null)
setTotalDuration(0)
setActiveRowIndices(new Set())
setReviewedRows(new Set())
setSkippedRows(new Set())
setCorrectedMap(new Map())
try {
const res = await fetch(
`${KLAUSUR_API}/api/v1/ocr-pipeline/sessions/${sessionId}/llm-review?stream=true`,
{ method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({}) },
)
if (!res.ok) {
const data = await res.json().catch(() => ({}))
throw new Error(data.detail || `HTTP ${res.status}`)
}
const reader = res.body!.getReader()
const decoder = new TextDecoder()
let buffer = ''
let allChanges: LlmChange[] = []
let allReviewed = new Set<number>()
let allSkipped = new Set<number>()
let cMap = new Map<number, LlmChange[]>()
while (true) {
const { done, value } = await reader.read()
if (done) break
buffer += decoder.decode(value, { stream: true })
while (buffer.includes('\n\n')) {
const idx = buffer.indexOf('\n\n')
const chunk = buffer.slice(0, idx).trim()
buffer = buffer.slice(idx + 2)
if (!chunk.startsWith('data: ')) continue
const dataStr = chunk.slice(6)
let event: any
try { event = JSON.parse(dataStr) } catch { continue }
if (event.type === 'meta') {
setMeta({
total_entries: event.total_entries,
to_review: event.to_review,
skipped: event.skipped,
model: event.model,
skipped_indices: event.skipped_indices,
})
// Mark skipped rows
if (event.skipped_indices) {
allSkipped = new Set(event.skipped_indices)
setSkippedRows(allSkipped)
}
}
if (event.type === 'batch') {
const batchChanges: LlmChange[] = event.changes || []
const batchRows: number[] = event.entries_reviewed || []
// Update active rows (currently being reviewed)
setActiveRowIndices(new Set(batchRows))
// Accumulate changes
allChanges = [...allChanges, ...batchChanges]
setChanges(allChanges)
setProgress(event.progress)
// Update corrected map
for (const c of batchChanges) {
const existing = cMap.get(c.row_index) || []
existing.push(c)
cMap.set(c.row_index, [...existing])
}
setCorrectedMap(new Map(cMap))
// Mark batch rows as reviewed
for (const r of batchRows) {
allReviewed.add(r)
}
setReviewedRows(new Set(allReviewed))
// Scroll to active row in table
setTimeout(() => {
activeRowRef.current?.scrollIntoView({ behavior: 'smooth', block: 'center' })
}, 50)
}
if (event.type === 'complete') {
setActiveRowIndices(new Set())
setTotalDuration(event.duration_ms)
setAccepted(new Set(allChanges.map((_: LlmChange, i: number) => i)))
// Mark all non-skipped as reviewed
const allEntryIndices = vocabEntries.map((_: WordEntry, i: number) => i)
for (const i of allEntryIndices) {
if (!allSkipped.has(i)) allReviewed.add(i)
}
setReviewedRows(new Set(allReviewed))
setStatus('done')
}
if (event.type === 'error') {
throw new Error(event.detail || 'Unbekannter Fehler')
}
}
}
// If stream ended without complete event
if (allChanges.length === 0) {
setStatus('done')
}
} catch (e: unknown) {
const msg = e instanceof Error ? e.message : String(e)
setError(msg)
setStatus('error')
}
}, [sessionId, vocabEntries])
const toggleChange = (index: number) => {
setAccepted(prev => {
const next = new Set(prev)
if (next.has(index)) next.delete(index)
else next.add(index)
return next
})
}
const toggleAll = () => {
if (accepted.size === changes.length) {
setAccepted(new Set())
} else {
setAccepted(new Set(changes.map((_: LlmChange, i: number) => i)))
}
}
const applyChanges = useCallback(async () => {
if (!sessionId) return
setApplying(true)
try {
const res = await fetch(`${KLAUSUR_API}/api/v1/ocr-pipeline/sessions/${sessionId}/llm-review/apply`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ accepted_indices: Array.from(accepted) }),
})
if (!res.ok) {
const data = await res.json().catch(() => ({}))
throw new Error(data.detail || `HTTP ${res.status}`)
}
setStatus('applied')
} catch (e: unknown) {
setError(e instanceof Error ? e.message : String(e))
} finally {
setApplying(false)
}
}, [sessionId, accepted])
const getRowStatus = (rowIndex: number): RowStatus => {
if (activeRowIndices.has(rowIndex)) return 'active'
if (skippedRows.has(rowIndex)) return 'skipped'
if (correctedMap.has(rowIndex)) return 'corrected'
if (reviewedRows.has(rowIndex)) return 'reviewed'
return 'pending'
}
const dewarpedUrl = sessionId
? `${KLAUSUR_API}/api/v1/ocr-pipeline/sessions/${sessionId}/image/dewarped`
: ''
if (!sessionId) {
return <div className="text-center py-12 text-gray-400">Bitte zuerst eine Session auswaehlen.</div>
}
// --- Loading session data ---
if (status === 'loading' || status === 'idle') {
return (
<div className="flex items-center gap-3 justify-center py-12">
<div className="animate-spin rounded-full h-5 w-5 border-b-2 border-teal-500" />
<span className="text-gray-500">Session-Daten werden geladen...</span>
</div>
)
}
// --- Error ---
if (status === 'error') {
return (
<div className="flex flex-col items-center justify-center py-12 text-center">
<div className="text-5xl mb-4"></div>
<h3 className="text-lg font-medium text-red-600 dark:text-red-400 mb-2">Fehler bei LLM-Korrektur</h3>
<p className="text-sm text-gray-500 dark:text-gray-400 max-w-lg mb-4">{error}</p>
<div className="flex gap-3">
<button onClick={() => { setError(''); loadSessionData() }}
className="px-5 py-2 bg-teal-600 text-white rounded-lg hover:bg-teal-700 transition-colors text-sm">
Erneut versuchen
</button>
<button onClick={onNext}
className="px-5 py-2 bg-gray-200 dark:bg-gray-700 text-gray-700 dark:text-gray-300 rounded-lg hover:bg-gray-300 dark:hover:bg-gray-600 transition-colors text-sm">
Ueberspringen
</button>
</div>
</div>
)
}
// --- Applied ---
if (status === 'applied') {
return (
<div className="flex flex-col items-center justify-center py-12 text-center">
<div className="text-5xl mb-4"></div>
<h3 className="text-lg font-medium text-gray-700 dark:text-gray-300 mb-2">Korrekturen uebernommen</h3>
<p className="text-sm text-gray-500 dark:text-gray-400 mb-6">
{accepted.size} von {changes.length} Korrekturen wurden angewendet.
</p>
<button onClick={onNext}
className="px-6 py-2.5 bg-teal-600 text-white rounded-lg hover:bg-teal-700 transition-colors font-medium">
Weiter
</button>
</div>
)
}
// Active entry for highlighting on image
const activeEntry = vocabEntries.find((_: WordEntry, i: number) => activeRowIndices.has(i))
const pct = progress ? Math.round((progress.current / progress.total) * 100) : 0
// --- Ready / Running / Done: 2-column layout ---
return (
<div className="space-y-4">
{/* Header */}
<div className="flex items-center justify-between">
<div>
<h3 className="text-base font-medium text-gray-700 dark:text-gray-300">
Schritt 6: LLM-Korrektur
</h3>
<p className="text-xs text-gray-400 mt-0.5">
{status === 'ready' && `${vocabEntries.length} Eintraege bereit zur Pruefung`}
{status === 'running' && meta && `${meta.model} · ${meta.to_review} zu pruefen, ${meta.skipped} uebersprungen`}
{status === 'done' && (
<>
{changes.length} Korrektur{changes.length !== 1 ? 'en' : ''} gefunden
{meta && <> · {meta.skipped} uebersprungen</>}
{' '}· {totalDuration}ms · {meta?.model}
</>
)}
</p>
</div>
<div className="flex items-center gap-2">
{status === 'ready' && (
<button onClick={runReview}
className="px-5 py-2 bg-teal-600 text-white rounded-lg hover:bg-teal-700 transition-colors text-sm font-medium">
LLM-Korrektur starten
</button>
)}
{status === 'running' && (
<div className="flex items-center gap-2 text-sm text-teal-600 dark:text-teal-400">
<div className="animate-spin rounded-full h-4 w-4 border-b-2 border-teal-500" />
{progress ? `${progress.current}/${progress.total}` : 'Startet...'}
</div>
)}
{status === 'done' && changes.length > 0 && (
<button onClick={toggleAll}
className="text-xs px-3 py-1.5 border border-gray-300 dark:border-gray-600 rounded-lg hover:bg-gray-50 dark:hover:bg-gray-700 transition-colors text-gray-600 dark:text-gray-400">
{accepted.size === changes.length ? 'Keine' : 'Alle'} auswaehlen
</button>
)}
</div>
</div>
{/* Progress bar (while running) */}
{status === 'running' && progress && (
<div className="space-y-1">
<div className="flex justify-between text-xs text-gray-400">
<span>{progress.current} / {progress.total} Eintraege geprueft</span>
<span>{pct}%</span>
</div>
<div className="w-full bg-gray-200 dark:bg-gray-700 rounded-full h-2">
<div className="bg-teal-500 h-2 rounded-full transition-all duration-500" style={{ width: `${pct}%` }} />
</div>
</div>
)}
{/* 2-column layout: Image + Table */}
<div className="grid grid-cols-3 gap-4">
{/* Left: Dewarped Image with highlight overlay */}
<div className="col-span-1">
<div className="text-xs font-medium text-gray-500 dark:text-gray-400 mb-1">
Originalbild
</div>
<div className="border rounded-lg overflow-hidden dark:border-gray-700 bg-gray-50 dark:bg-gray-900 relative sticky top-4">
{/* eslint-disable-next-line @next/next/no-img-element */}
<img
src={dewarpedUrl}
alt="Dewarped"
className="w-full h-auto"
onLoad={(e) => {
const img = e.target as HTMLImageElement
setImageNaturalSize({ w: img.naturalWidth, h: img.naturalHeight })
}}
/>
{/* Highlight overlay for active row */}
{activeEntry?.bbox && (
<div
className="absolute border-2 border-yellow-400 bg-yellow-400/20 pointer-events-none animate-pulse"
style={{
left: `${activeEntry.bbox.x}%`,
top: `${activeEntry.bbox.y}%`,
width: `${activeEntry.bbox.w}%`,
height: `${activeEntry.bbox.h}%`,
}}
/>
)}
</div>
</div>
{/* Right: Full vocabulary table */}
<div className="col-span-2" ref={tableRef}>
<div className="text-xs font-medium text-gray-500 dark:text-gray-400 mb-1">
Vokabeltabelle ({vocabEntries.length} Eintraege)
</div>
<div className="border border-gray-200 dark:border-gray-700 rounded-lg overflow-hidden">
<div className="max-h-[70vh] overflow-y-auto">
<table className="w-full text-sm">
<thead className="sticky top-0 z-10">
<tr className="bg-gray-50 dark:bg-gray-800 border-b border-gray-200 dark:border-gray-700">
<th className="px-2 py-2 text-left text-gray-500 dark:text-gray-400 font-medium w-10">#</th>
<th className="px-2 py-2 text-left text-gray-500 dark:text-gray-400 font-medium">EN</th>
<th className="px-2 py-2 text-left text-gray-500 dark:text-gray-400 font-medium">DE</th>
<th className="px-2 py-2 text-left text-gray-500 dark:text-gray-400 font-medium">Beispiel</th>
<th className="px-2 py-2 text-center text-gray-500 dark:text-gray-400 font-medium w-16">Status</th>
</tr>
</thead>
<tbody>
{vocabEntries.map((entry, idx) => {
const rowStatus = getRowStatus(idx)
const rowChanges = correctedMap.get(idx)
const rowBg = {
pending: '',
active: 'bg-yellow-50 dark:bg-yellow-900/20',
reviewed: '',
corrected: 'bg-teal-50/50 dark:bg-teal-900/10',
skipped: 'bg-gray-50 dark:bg-gray-800/50',
}[rowStatus]
return (
<tr
key={idx}
ref={rowStatus === 'active' ? activeRowRef : undefined}
className={`border-b border-gray-100 dark:border-gray-700/50 ${rowBg} ${
rowStatus === 'active' ? 'ring-1 ring-yellow-400 ring-inset' : ''
}`}
>
<td className="px-2 py-1.5 text-gray-400 font-mono text-xs">{idx}</td>
<td className="px-2 py-1.5">
<CellContent text={entry.english} field="english" rowChanges={rowChanges} />
</td>
<td className="px-2 py-1.5">
<CellContent text={entry.german} field="german" rowChanges={rowChanges} />
</td>
<td className="px-2 py-1.5 text-xs">
<CellContent text={entry.example} field="example" rowChanges={rowChanges} />
</td>
<td className="px-2 py-1.5 text-center">
<StatusIcon status={rowStatus} />
</td>
</tr>
)
})}
</tbody>
</table>
</div>
</div>
</div>
</div>
{/* Done state: summary + actions */}
{status === 'done' && (
<div className="space-y-4">
{/* Summary */}
<div className="bg-gray-50 dark:bg-gray-800/50 rounded-lg p-3 text-xs text-gray-500 dark:text-gray-400">
{changes.length === 0 ? (
<span>Keine Korrekturen noetig alle Eintraege sind korrekt.</span>
) : (
<span>
{changes.length} Korrektur{changes.length !== 1 ? 'en' : ''} gefunden ·{' '}
{accepted.size} ausgewaehlt ·{' '}
{meta?.skipped || 0} uebersprungen (Lautschrift) ·{' '}
{totalDuration}ms
</span>
)}
</div>
{/* Corrections detail list (if any) */}
{changes.length > 0 && (
<div className="border border-gray-200 dark:border-gray-700 rounded-lg overflow-hidden">
<div className="bg-gray-50 dark:bg-gray-800 px-3 py-2 border-b border-gray-200 dark:border-gray-700">
<span className="text-xs font-medium text-gray-600 dark:text-gray-400">
Korrekturvorschlaege ({accepted.size}/{changes.length} ausgewaehlt)
</span>
</div>
<table className="w-full text-sm">
<thead>
<tr className="bg-gray-50/50 dark:bg-gray-800/50 border-b border-gray-200 dark:border-gray-700">
<th className="w-10 px-3 py-1.5 text-center">
<input type="checkbox" checked={accepted.size === changes.length} onChange={toggleAll}
className="rounded border-gray-300 dark:border-gray-600" />
</th>
<th className="px-2 py-1.5 text-left text-gray-500 dark:text-gray-400 font-medium text-xs">Zeile</th>
<th className="px-2 py-1.5 text-left text-gray-500 dark:text-gray-400 font-medium text-xs">Feld</th>
<th className="px-2 py-1.5 text-left text-gray-500 dark:text-gray-400 font-medium text-xs">Vorher</th>
<th className="px-2 py-1.5 text-left text-gray-500 dark:text-gray-400 font-medium text-xs">Nachher</th>
</tr>
</thead>
<tbody>
{changes.map((change, idx) => (
<tr key={idx} className={`border-b border-gray-100 dark:border-gray-700/50 ${
accepted.has(idx) ? 'bg-teal-50/50 dark:bg-teal-900/10' : ''
}`}>
<td className="px-3 py-1.5 text-center">
<input type="checkbox" checked={accepted.has(idx)} onChange={() => toggleChange(idx)}
className="rounded border-gray-300 dark:border-gray-600" />
</td>
<td className="px-2 py-1.5 text-gray-500 dark:text-gray-400 font-mono text-xs">R{change.row_index}</td>
<td className="px-2 py-1.5">
<span className="text-xs px-1.5 py-0.5 rounded bg-gray-100 dark:bg-gray-700 text-gray-600 dark:text-gray-400">
{FIELD_LABELS[change.field] || change.field}
</span>
</td>
<td className="px-2 py-1.5"><span className="line-through text-red-500 dark:text-red-400 text-xs">{change.old}</span></td>
<td className="px-2 py-1.5"><span className="text-green-600 dark:text-green-400 font-medium text-xs">{change.new}</span></td>
</tr>
))}
</tbody>
</table>
</div>
)}
{/* Actions */}
<div className="flex items-center justify-between pt-2">
<p className="text-xs text-gray-400">
{changes.length > 0 ? `${accepted.size} von ${changes.length} ausgewaehlt` : ''}
</p>
<div className="flex gap-3">
{changes.length > 0 && (
<button onClick={onNext}
className="px-4 py-2 text-sm border border-gray-300 dark:border-gray-600 rounded-lg hover:bg-gray-50 dark:hover:bg-gray-700 transition-colors text-gray-600 dark:text-gray-400">
Alle ablehnen
</button>
)}
{changes.length > 0 ? (
<button onClick={applyChanges} disabled={applying || accepted.size === 0}
className="px-5 py-2 text-sm bg-teal-600 text-white rounded-lg hover:bg-teal-700 disabled:opacity-50 disabled:cursor-not-allowed transition-colors font-medium">
{applying ? 'Wird uebernommen...' : `${accepted.size} Korrektur${accepted.size !== 1 ? 'en' : ''} uebernehmen`}
</button>
) : (
<button onClick={onNext}
className="px-6 py-2.5 bg-teal-600 text-white rounded-lg hover:bg-teal-700 transition-colors font-medium">
Weiter
</button>
)}
</div>
</div>
</div>
)}
</div>
)
}
/** Cell content with inline diff for corrections */
function CellContent({ text, field, rowChanges }: {
text: string
field: string
rowChanges?: LlmChange[]
}) {
const change = rowChanges?.find(c => c.field === field)
if (!text && !change) {
return <span className="text-gray-300 dark:text-gray-600">&mdash;</span>
}
if (change) {
return (
<span>
<span className="line-through text-red-400 dark:text-red-500 text-xs mr-1">{change.old}</span>
<span className="text-green-600 dark:text-green-400 font-medium text-xs">{change.new}</span>
</span>
)
}
return <span className="text-gray-700 dark:text-gray-300 text-xs">{text}</span>
}
/** Status icon for each row */
function StatusIcon({ status }: { status: RowStatus }) {
switch (status) {
case 'pending':
return <span className="text-gray-300 dark:text-gray-600 text-xs"></span>
case 'active':
return (
<span className="inline-block w-3 h-3 rounded-full bg-yellow-400 animate-pulse" title="Wird geprueft" />
)
case 'reviewed':
return (
<svg className="w-4 h-4 text-green-500 inline-block" fill="none" viewBox="0 0 24 24" stroke="currentColor" strokeWidth={2}>
<path strokeLinecap="round" strokeLinejoin="round" d="M5 13l4 4L19 7" />
</svg>
)
case 'corrected':
return (
<span className="inline-flex items-center px-1.5 py-0.5 rounded text-[10px] font-medium bg-teal-100 dark:bg-teal-900/30 text-teal-700 dark:text-teal-400">
korr.
</span>
)
case 'skipped':
return (
<span className="inline-flex items-center px-1.5 py-0.5 rounded text-[10px] font-medium bg-gray-100 dark:bg-gray-700 text-gray-500 dark:text-gray-400">
skip
</span>
)
}
}

View File

@@ -0,0 +1,505 @@
'use client'
import { useCallback, useEffect, useMemo, useRef, useState } from 'react'
import type { GridResult, GridCell, WordEntry } from '@/app/(admin)/ai/ocr-pipeline/types'
const KLAUSUR_API = '/klausur-api'
interface StepReconstructionProps {
sessionId: string | null
onNext: () => void
}
interface EditableCell {
cellId: string
text: string
originalText: string
bboxPct: { x: number; y: number; w: number; h: number }
colType: string
rowIndex: number
colIndex: number
}
type UndoAction = { cellId: string; oldText: string; newText: string }
export function StepReconstruction({ sessionId, onNext }: StepReconstructionProps) {
const [status, setStatus] = useState<'loading' | 'ready' | 'saving' | 'saved' | 'error'>('loading')
const [error, setError] = useState('')
const [cells, setCells] = useState<EditableCell[]>([])
const [editedTexts, setEditedTexts] = useState<Map<string, string>>(new Map())
const [zoom, setZoom] = useState(100)
const [imageNaturalH, setImageNaturalH] = useState(0)
const [showEmptyHighlight, setShowEmptyHighlight] = useState(true)
// Undo/Redo stacks
const [undoStack, setUndoStack] = useState<UndoAction[]>([])
const [redoStack, setRedoStack] = useState<UndoAction[]>([])
// All cells including empty ones (for empty field highlighting)
const [allCells, setAllCells] = useState<EditableCell[]>([])
const containerRef = useRef<HTMLDivElement>(null)
const imageRef = useRef<HTMLImageElement>(null)
// Load session data on mount
useEffect(() => {
if (!sessionId) return
loadSessionData()
// eslint-disable-next-line react-hooks/exhaustive-deps
}, [sessionId])
// Track image natural height for font scaling
const handleImageLoad = useCallback(() => {
if (imageRef.current) {
setImageNaturalH(imageRef.current.naturalHeight)
}
}, [])
const loadSessionData = async () => {
if (!sessionId) return
setStatus('loading')
try {
const res = await fetch(`${KLAUSUR_API}/api/v1/ocr-pipeline/sessions/${sessionId}`)
if (!res.ok) throw new Error(`HTTP ${res.status}`)
const data = await res.json()
const wordResult: GridResult | undefined = data.word_result
if (!wordResult) {
setError('Keine Worterkennungsdaten gefunden. Bitte zuerst Schritt 5 abschliessen.')
setStatus('error')
return
}
// Build editable cells from grid cells
const gridCells: GridCell[] = wordResult.cells || []
const allEditableCells: EditableCell[] = gridCells.map(c => ({
cellId: c.cell_id,
text: c.text,
originalText: c.text,
bboxPct: c.bbox_pct,
colType: c.col_type,
rowIndex: c.row_index,
colIndex: c.col_index,
}))
setAllCells(allEditableCells)
setCells(allEditableCells.filter(c => c.text.trim() !== ''))
setEditedTexts(new Map())
setUndoStack([])
setRedoStack([])
setStatus('ready')
} catch (e: unknown) {
setError(e instanceof Error ? e.message : String(e))
setStatus('error')
}
}
const handleTextChange = useCallback((cellId: string, newText: string) => {
setEditedTexts(prev => {
const oldText = prev.get(cellId)
const cell = cells.find(c => c.cellId === cellId)
const prevText = oldText ?? cell?.text ?? ''
// Push to undo stack
setUndoStack(stack => [...stack, { cellId, oldText: prevText, newText }])
setRedoStack([]) // Clear redo on new edit
const next = new Map(prev)
next.set(cellId, newText)
return next
})
}, [cells])
const undo = useCallback(() => {
setUndoStack(stack => {
if (stack.length === 0) return stack
const action = stack[stack.length - 1]
const newStack = stack.slice(0, -1)
setRedoStack(rs => [...rs, action])
setEditedTexts(prev => {
const next = new Map(prev)
next.set(action.cellId, action.oldText)
return next
})
return newStack
})
}, [])
const redo = useCallback(() => {
setRedoStack(stack => {
if (stack.length === 0) return stack
const action = stack[stack.length - 1]
const newStack = stack.slice(0, -1)
setUndoStack(us => [...us, action])
setEditedTexts(prev => {
const next = new Map(prev)
next.set(action.cellId, action.newText)
return next
})
return newStack
})
}, [])
const resetCell = useCallback((cellId: string) => {
const cell = cells.find(c => c.cellId === cellId)
if (!cell) return
setEditedTexts(prev => {
const next = new Map(prev)
next.delete(cellId)
return next
})
}, [cells])
// Global keyboard shortcuts for undo/redo
useEffect(() => {
const handler = (e: KeyboardEvent) => {
if ((e.metaKey || e.ctrlKey) && e.key === 'z') {
e.preventDefault()
if (e.shiftKey) {
redo()
} else {
undo()
}
}
}
document.addEventListener('keydown', handler)
return () => document.removeEventListener('keydown', handler)
}, [undo, redo])
const getDisplayText = useCallback((cell: EditableCell): string => {
return editedTexts.get(cell.cellId) ?? cell.text
}, [editedTexts])
const isEdited = useCallback((cell: EditableCell): boolean => {
const edited = editedTexts.get(cell.cellId)
return edited !== undefined && edited !== cell.originalText
}, [editedTexts])
const changedCount = useMemo(() => {
let count = 0
for (const cell of cells) {
if (isEdited(cell)) count++
}
return count
}, [cells, isEdited])
// Identify empty required cells (EN or DE columns with no text)
const emptyCellIds = useMemo(() => {
const required = new Set(['column_en', 'column_de'])
const ids = new Set<string>()
for (const cell of allCells) {
if (required.has(cell.colType) && !cell.text.trim()) {
ids.add(cell.cellId)
}
}
return ids
}, [allCells])
// Sort cells for tab navigation: by row, then by column
const sortedCellIds = useMemo(() => {
return [...cells]
.sort((a, b) => a.rowIndex !== b.rowIndex ? a.rowIndex - b.rowIndex : a.colIndex - b.colIndex)
.map(c => c.cellId)
}, [cells])
const handleKeyDown = useCallback((e: React.KeyboardEvent, cellId: string) => {
if (e.key === 'Tab') {
e.preventDefault()
const idx = sortedCellIds.indexOf(cellId)
const nextIdx = e.shiftKey ? idx - 1 : idx + 1
if (nextIdx >= 0 && nextIdx < sortedCellIds.length) {
const nextId = sortedCellIds[nextIdx]
const el = document.getElementById(`cell-${nextId}`)
el?.focus()
}
}
}, [sortedCellIds])
const saveReconstruction = useCallback(async () => {
if (!sessionId) return
setStatus('saving')
try {
const cellUpdates = Array.from(editedTexts.entries())
.filter(([cellId, text]) => {
const cell = cells.find(c => c.cellId === cellId)
return cell && text !== cell.originalText
})
.map(([cellId, text]) => ({ cell_id: cellId, text }))
if (cellUpdates.length === 0) {
// Nothing changed, just advance
setStatus('saved')
return
}
const res = await fetch(`${KLAUSUR_API}/api/v1/ocr-pipeline/sessions/${sessionId}/reconstruction`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ cells: cellUpdates }),
})
if (!res.ok) {
const data = await res.json().catch(() => ({}))
throw new Error(data.detail || `HTTP ${res.status}`)
}
setStatus('saved')
} catch (e: unknown) {
setError(e instanceof Error ? e.message : String(e))
setStatus('error')
}
}, [sessionId, editedTexts, cells])
const dewarpedUrl = sessionId
? `${KLAUSUR_API}/api/v1/ocr-pipeline/sessions/${sessionId}/image/dewarped`
: ''
const colTypeColor = (colType: string): string => {
const colors: Record<string, string> = {
column_en: 'border-blue-400/40 focus:border-blue-500',
column_de: 'border-green-400/40 focus:border-green-500',
column_example: 'border-orange-400/40 focus:border-orange-500',
column_text: 'border-purple-400/40 focus:border-purple-500',
}
return colors[colType] || 'border-gray-400/40 focus:border-gray-500'
}
// Font size based on image natural height (not container) scaled by zoom
const getFontSize = useCallback((bboxH: number): number => {
const baseH = imageNaturalH || 800
const px = (bboxH / 100) * baseH * 0.55
return Math.max(8, Math.min(18, px * (zoom / 100)))
}, [imageNaturalH, zoom])
if (!sessionId) {
return <div className="text-center py-12 text-gray-400">Bitte zuerst eine Session auswaehlen.</div>
}
if (status === 'loading') {
return (
<div className="flex items-center gap-3 justify-center py-12">
<div className="animate-spin rounded-full h-5 w-5 border-b-2 border-teal-500" />
<span className="text-gray-500">Rekonstruktionsdaten werden geladen...</span>
</div>
)
}
if (status === 'error') {
return (
<div className="flex flex-col items-center justify-center py-12 text-center">
<div className="text-5xl mb-4">&#x26A0;&#xFE0F;</div>
<h3 className="text-lg font-medium text-red-600 dark:text-red-400 mb-2">Fehler</h3>
<p className="text-sm text-gray-500 dark:text-gray-400 max-w-lg mb-4">{error}</p>
<div className="flex gap-3">
<button onClick={() => { setError(''); loadSessionData() }}
className="px-5 py-2 bg-teal-600 text-white rounded-lg hover:bg-teal-700 transition-colors text-sm">
Erneut versuchen
</button>
<button onClick={onNext}
className="px-5 py-2 bg-gray-200 dark:bg-gray-700 text-gray-700 dark:text-gray-300 rounded-lg hover:bg-gray-300 dark:hover:bg-gray-600 transition-colors text-sm">
Ueberspringen &rarr;
</button>
</div>
</div>
)
}
if (status === 'saved') {
return (
<div className="flex flex-col items-center justify-center py-12 text-center">
<div className="text-5xl mb-4">&#x2705;</div>
<h3 className="text-lg font-medium text-gray-700 dark:text-gray-300 mb-2">Rekonstruktion gespeichert</h3>
<p className="text-sm text-gray-500 dark:text-gray-400 mb-6">
{changedCount > 0 ? `${changedCount} Zellen wurden aktualisiert.` : 'Keine Aenderungen vorgenommen.'}
</p>
<button onClick={onNext}
className="px-6 py-2.5 bg-teal-600 text-white rounded-lg hover:bg-teal-700 transition-colors font-medium">
Weiter &rarr;
</button>
</div>
)
}
return (
<div className="space-y-3">
{/* Toolbar */}
<div className="flex items-center justify-between bg-white dark:bg-gray-800 rounded-lg border border-gray-200 dark:border-gray-700 px-3 py-2">
<div className="flex items-center gap-2">
<h3 className="text-sm font-medium text-gray-700 dark:text-gray-300">
Schritt 7: Rekonstruktion
</h3>
<span className="text-xs text-gray-400">
{cells.length} Zellen &middot; {changedCount} geaendert
{emptyCellIds.size > 0 && showEmptyHighlight && (
<span className="text-red-400 ml-1">&middot; {emptyCellIds.size} leer</span>
)}
</span>
</div>
<div className="flex items-center gap-2">
{/* Undo/Redo */}
<button
onClick={undo}
disabled={undoStack.length === 0}
className="px-2 py-1 text-xs border border-gray-300 dark:border-gray-600 rounded hover:bg-gray-50 dark:hover:bg-gray-700 disabled:opacity-30"
title="Rueckgaengig (Ctrl+Z)"
>
&#x21A9;
</button>
<button
onClick={redo}
disabled={redoStack.length === 0}
className="px-2 py-1 text-xs border border-gray-300 dark:border-gray-600 rounded hover:bg-gray-50 dark:hover:bg-gray-700 disabled:opacity-30"
title="Wiederholen (Ctrl+Shift+Z)"
>
&#x21AA;
</button>
<div className="w-px h-5 bg-gray-300 dark:bg-gray-600 mx-1" />
{/* Empty field toggle */}
<button
onClick={() => setShowEmptyHighlight(v => !v)}
className={`px-2 py-1 text-xs border rounded transition-colors ${
showEmptyHighlight
? 'border-red-300 bg-red-50 text-red-600 dark:border-red-700 dark:bg-red-900/30 dark:text-red-400'
: 'border-gray-300 dark:border-gray-600 hover:bg-gray-50 dark:hover:bg-gray-700'
}`}
title="Leere Pflichtfelder markieren"
>
Leer
</button>
<div className="w-px h-5 bg-gray-300 dark:bg-gray-600 mx-1" />
{/* Zoom controls */}
<button
onClick={() => setZoom(z => Math.max(50, z - 25))}
className="px-2 py-1 text-xs border border-gray-300 dark:border-gray-600 rounded hover:bg-gray-50 dark:hover:bg-gray-700"
>
&minus;
</button>
<span className="text-xs text-gray-500 w-10 text-center">{zoom}%</span>
<button
onClick={() => setZoom(z => Math.min(200, z + 25))}
className="px-2 py-1 text-xs border border-gray-300 dark:border-gray-600 rounded hover:bg-gray-50 dark:hover:bg-gray-700"
>
+
</button>
<button
onClick={() => setZoom(100)}
className="px-2 py-1 text-xs border border-gray-300 dark:border-gray-600 rounded hover:bg-gray-50 dark:hover:bg-gray-700"
>
Fit
</button>
<div className="w-px h-5 bg-gray-300 dark:bg-gray-600 mx-1" />
<button
onClick={saveReconstruction}
disabled={status === 'saving'}
className="px-4 py-1.5 text-xs bg-teal-600 text-white rounded-lg hover:bg-teal-700 disabled:opacity-50 transition-colors font-medium"
>
{status === 'saving' ? 'Speichert...' : 'Speichern'}
</button>
</div>
</div>
{/* Reconstruction canvas */}
<div className="border rounded-lg overflow-auto dark:border-gray-700 bg-gray-100 dark:bg-gray-900" style={{ maxHeight: '75vh' }}>
<div
ref={containerRef}
className="relative inline-block"
style={{ transform: `scale(${zoom / 100})`, transformOrigin: 'top left' }}
>
{/* Background image at reduced opacity */}
{/* eslint-disable-next-line @next/next/no-img-element */}
<img
ref={imageRef}
src={dewarpedUrl}
alt="Dewarped"
className="block"
style={{ opacity: 0.3 }}
onLoad={handleImageLoad}
/>
{/* Empty field markers */}
{showEmptyHighlight && allCells
.filter(c => emptyCellIds.has(c.cellId))
.map(cell => (
<div
key={`empty-${cell.cellId}`}
className="absolute border-2 border-dashed border-red-400/60 rounded pointer-events-none"
style={{
left: `${cell.bboxPct.x}%`,
top: `${cell.bboxPct.y}%`,
width: `${cell.bboxPct.w}%`,
height: `${cell.bboxPct.h}%`,
}}
/>
))}
{/* Editable text fields at bbox positions */}
{cells.map((cell) => {
const displayText = getDisplayText(cell)
const edited = isEdited(cell)
return (
<div key={cell.cellId} className="absolute group" style={{
left: `${cell.bboxPct.x}%`,
top: `${cell.bboxPct.y}%`,
width: `${cell.bboxPct.w}%`,
height: `${cell.bboxPct.h}%`,
}}>
<input
id={`cell-${cell.cellId}`}
type="text"
value={displayText}
onChange={(e) => handleTextChange(cell.cellId, e.target.value)}
onKeyDown={(e) => handleKeyDown(e, cell.cellId)}
className={`w-full h-full bg-transparent text-black dark:text-white border px-0.5 outline-none transition-colors ${
colTypeColor(cell.colType)
} ${edited ? 'border-green-500 bg-green-50/30 dark:bg-green-900/20' : ''}`}
style={{
fontSize: `${getFontSize(cell.bboxPct.h)}px`,
lineHeight: '1',
}}
title={`${cell.cellId} (${cell.colType})`}
/>
{/* Per-cell reset button (X) — only shown for edited cells on hover */}
{edited && (
<button
onClick={() => resetCell(cell.cellId)}
className="absolute -top-1 -right-1 w-4 h-4 bg-red-500 text-white rounded-full text-[9px] leading-none opacity-0 group-hover:opacity-100 transition-opacity flex items-center justify-center"
title="Zuruecksetzen"
>
&times;
</button>
)}
</div>
)
})}
</div>
</div>
{/* Bottom action */}
<div className="flex justify-end">
<button
onClick={() => {
if (changedCount > 0) {
saveReconstruction()
} else {
onNext()
}
}}
className="px-6 py-2.5 bg-teal-600 text-white rounded-lg hover:bg-teal-700 transition-colors font-medium text-sm"
>
{changedCount > 0 ? 'Speichern & Weiter \u2192' : 'Weiter \u2192'}
</button>
</div>
</div>
)
}

View File

@@ -0,0 +1,263 @@
'use client'
import { useCallback, useEffect, useState } from 'react'
import type { RowResult, RowGroundTruth } from '@/app/(admin)/ai/ocr-pipeline/types'
const KLAUSUR_API = '/klausur-api'
interface StepRowDetectionProps {
sessionId: string | null
onNext: () => void
}
export function StepRowDetection({ sessionId, onNext }: StepRowDetectionProps) {
const [rowResult, setRowResult] = useState<RowResult | null>(null)
const [detecting, setDetecting] = useState(false)
const [error, setError] = useState<string | null>(null)
const [gtNotes, setGtNotes] = useState('')
const [gtSaved, setGtSaved] = useState(false)
useEffect(() => {
if (!sessionId) return
const fetchSession = async () => {
try {
const res = await fetch(`${KLAUSUR_API}/api/v1/ocr-pipeline/sessions/${sessionId}`)
if (res.ok) {
const info = await res.json()
if (info.row_result) {
setRowResult(info.row_result)
return
}
}
} catch (e) {
console.error('Failed to fetch session info:', e)
}
// No cached result — run auto
runAutoDetection()
}
fetchSession()
// eslint-disable-next-line react-hooks/exhaustive-deps
}, [sessionId])
const runAutoDetection = useCallback(async () => {
if (!sessionId) return
setDetecting(true)
setError(null)
try {
const res = await fetch(`${KLAUSUR_API}/api/v1/ocr-pipeline/sessions/${sessionId}/rows`, {
method: 'POST',
})
if (!res.ok) {
const err = await res.json().catch(() => ({ detail: res.statusText }))
throw new Error(err.detail || 'Zeilenerkennung fehlgeschlagen')
}
const data: RowResult = await res.json()
setRowResult(data)
} catch (e) {
setError(e instanceof Error ? e.message : 'Unbekannter Fehler')
} finally {
setDetecting(false)
}
}, [sessionId])
const handleGroundTruth = useCallback(async (isCorrect: boolean) => {
if (!sessionId) return
const gt: RowGroundTruth = {
is_correct: isCorrect,
notes: gtNotes || undefined,
}
try {
await fetch(`${KLAUSUR_API}/api/v1/ocr-pipeline/sessions/${sessionId}/ground-truth/rows`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify(gt),
})
setGtSaved(true)
} catch (e) {
console.error('Ground truth save failed:', e)
}
}, [sessionId, gtNotes])
if (!sessionId) {
return (
<div className="flex flex-col items-center justify-center py-16 text-center">
<div className="text-5xl mb-4">📏</div>
<h3 className="text-lg font-medium text-gray-700 dark:text-gray-300 mb-2">
Schritt 4: Zeilenerkennung
</h3>
<p className="text-gray-500 dark:text-gray-400 max-w-md">
Bitte zuerst Schritte 1-3 abschliessen.
</p>
</div>
)
}
const overlayUrl = `${KLAUSUR_API}/api/v1/ocr-pipeline/sessions/${sessionId}/image/rows-overlay`
const dewarpedUrl = `${KLAUSUR_API}/api/v1/ocr-pipeline/sessions/${sessionId}/image/dewarped`
const rowTypeColors: Record<string, string> = {
header: 'bg-gray-200 dark:bg-gray-600 text-gray-700 dark:text-gray-300',
content: 'bg-blue-100 dark:bg-blue-900/30 text-blue-700 dark:text-blue-300',
footer: 'bg-gray-200 dark:bg-gray-600 text-gray-700 dark:text-gray-300',
}
return (
<div className="space-y-4">
{/* Loading */}
{detecting && (
<div className="flex items-center gap-2 text-teal-600 dark:text-teal-400 text-sm">
<div className="animate-spin w-4 h-4 border-2 border-teal-500 border-t-transparent rounded-full" />
Zeilenerkennung laeuft...
</div>
)}
{/* Images: overlay vs clean */}
<div className="grid grid-cols-2 gap-4">
<div>
<div className="text-xs font-medium text-gray-500 dark:text-gray-400 mb-1">
Mit Zeilen-Overlay
</div>
<div className="border rounded-lg overflow-hidden dark:border-gray-700 bg-gray-50 dark:bg-gray-900">
{rowResult ? (
// eslint-disable-next-line @next/next/no-img-element
<img
src={`${overlayUrl}?t=${Date.now()}`}
alt="Zeilen-Overlay"
className="w-full h-auto"
/>
) : (
<div className="aspect-[3/4] flex items-center justify-center text-gray-400 text-sm">
{detecting ? 'Erkenne Zeilen...' : 'Keine Daten'}
</div>
)}
</div>
</div>
<div>
<div className="text-xs font-medium text-gray-500 dark:text-gray-400 mb-1">
Entzerrtes Bild
</div>
<div className="border rounded-lg overflow-hidden dark:border-gray-700 bg-gray-50 dark:bg-gray-900">
{/* eslint-disable-next-line @next/next/no-img-element */}
<img
src={dewarpedUrl}
alt="Entzerrt"
className="w-full h-auto"
/>
</div>
</div>
</div>
{/* Row summary */}
{rowResult && (
<div className="bg-white dark:bg-gray-800 rounded-xl border border-gray-200 dark:border-gray-700 p-4 space-y-3">
<div className="flex items-center justify-between">
<h4 className="text-sm font-medium text-gray-700 dark:text-gray-300">
Ergebnis: {rowResult.total_rows} Zeilen erkannt
</h4>
<span className="text-xs text-gray-400">
{rowResult.duration_seconds}s
</span>
</div>
{/* Type summary badges */}
<div className="flex gap-2">
{Object.entries(rowResult.summary).map(([type, count]) => (
<span
key={type}
className={`px-2 py-0.5 rounded text-xs font-medium ${rowTypeColors[type] || 'bg-gray-100 text-gray-600'}`}
>
{type}: {count}
</span>
))}
</div>
{/* Row list */}
<div className="max-h-64 overflow-y-auto space-y-1">
{rowResult.rows.map((row) => (
<div
key={row.index}
className={`flex items-center gap-3 px-3 py-1.5 rounded text-xs font-mono ${
row.row_type === 'header' || row.row_type === 'footer'
? 'bg-gray-50 dark:bg-gray-700/50 text-gray-500'
: 'text-gray-600 dark:text-gray-400'
}`}
>
<span className="w-8 text-right text-gray-400">R{row.index}</span>
<span className={`px-1.5 py-0.5 rounded text-[10px] uppercase font-semibold ${rowTypeColors[row.row_type] || ''}`}>
{row.row_type}
</span>
<span>y={row.y}</span>
<span>h={row.height}px</span>
<span>{row.word_count} Woerter</span>
{row.gap_before > 0 && (
<span className="text-gray-400">gap={row.gap_before}px</span>
)}
</div>
))}
</div>
</div>
)}
{/* Controls */}
{rowResult && (
<div className="bg-white dark:bg-gray-800 rounded-xl border border-gray-200 dark:border-gray-700 p-4 space-y-3">
<div className="flex items-center gap-3">
<button
onClick={() => runAutoDetection()}
disabled={detecting}
className="px-3 py-1.5 text-xs border rounded-lg hover:bg-gray-50 dark:hover:bg-gray-700 dark:border-gray-600 disabled:opacity-50"
>
Erneut erkennen
</button>
<div className="flex-1" />
{/* Ground truth */}
{!gtSaved ? (
<>
<input
type="text"
placeholder="Notizen (optional)"
value={gtNotes}
onChange={(e) => setGtNotes(e.target.value)}
className="px-2 py-1 text-xs border rounded dark:bg-gray-700 dark:border-gray-600 w-48"
/>
<button
onClick={() => handleGroundTruth(true)}
className="px-3 py-1.5 text-xs bg-green-600 text-white rounded-lg hover:bg-green-700"
>
Korrekt
</button>
<button
onClick={() => handleGroundTruth(false)}
className="px-3 py-1.5 text-xs bg-red-600 text-white rounded-lg hover:bg-red-700"
>
Fehlerhaft
</button>
</>
) : (
<span className="text-xs text-green-600 dark:text-green-400">
Ground Truth gespeichert
</span>
)}
<button
onClick={onNext}
className="px-4 py-1.5 text-xs bg-teal-600 text-white rounded-lg hover:bg-teal-700 font-medium"
>
Weiter
</button>
</div>
</div>
)}
{error && (
<div className="p-3 bg-red-50 dark:bg-red-900/20 text-red-600 dark:text-red-400 rounded-lg text-sm">
{error}
</div>
)}
</div>
)
}

File diff suppressed because it is too large Load Diff

View File

@@ -127,6 +127,15 @@ export const navigation: NavCategory[] = [
audience: ['Entwickler', 'Data Scientists', 'Lehrer'],
subgroup: 'KI-Werkzeuge',
},
{
id: 'ocr-pipeline',
name: 'OCR Pipeline',
href: '/ai/ocr-pipeline',
description: 'Schrittweise Seitenrekonstruktion',
purpose: 'Schrittweise Seitenrekonstruktion: Scan begradigen, Spalten erkennen, Woerter lokalisieren und die Seite Wort fuer Wort nachbauen. 6-Schritt-Pipeline mit Ground Truth Validierung.',
audience: ['Entwickler', 'Data Scientists'],
subgroup: 'KI-Werkzeuge',
},
{
id: 'test-quality',
name: 'Test Quality (BQAS)',

View File

@@ -2,6 +2,8 @@
const nextConfig = {
output: 'standalone',
reactStrictMode: true,
// Force unique build ID to bust browser caches on each deploy
generateBuildId: () => `build-${Date.now()}`,
// TODO: Remove after fixing type incompatibilities from restore
typescript: {
ignoreBuildErrors: true,

View File

@@ -8,6 +8,7 @@
"name": "breakpilot-admin-v2",
"version": "1.0.0",
"dependencies": {
"bpmn-js": "^18.0.1",
"jspdf": "^4.1.0",
"jszip": "^3.10.1",
"lucide-react": "^0.468.0",
@@ -15,6 +16,7 @@
"react": "^18.3.1",
"react-dom": "^18.3.1",
"reactflow": "^11.11.4",
"recharts": "^2.15.0",
"uuid": "^13.0.0"
},
"devDependencies": {
@@ -428,6 +430,16 @@
"node": ">=6.9.0"
}
},
"node_modules/@bpmn-io/diagram-js-ui": {
"version": "0.2.3",
"resolved": "https://registry.npmjs.org/@bpmn-io/diagram-js-ui/-/diagram-js-ui-0.2.3.tgz",
"integrity": "sha512-OGyjZKvGK8tHSZ0l7RfeKhilGoOGtFDcoqSGYkX0uhFlo99OVZ9Jn1K7TJGzcE9BdKwvA5Y5kGqHEhdTxHvFfw==",
"license": "MIT",
"dependencies": {
"htm": "^3.1.1",
"preact": "^10.11.2"
}
},
"node_modules/@csstools/color-helpers": {
"version": "5.1.0",
"resolved": "https://registry.npmjs.org/@csstools/color-helpers/-/color-helpers-5.1.0.tgz",
@@ -2996,6 +3008,39 @@
"url": "https://github.com/sponsors/sindresorhus"
}
},
"node_modules/bpmn-js": {
"version": "18.12.0",
"resolved": "https://registry.npmjs.org/bpmn-js/-/bpmn-js-18.12.0.tgz",
"integrity": "sha512-Dg2O+r7jpBwLgWGpManc7P4ZfZQfxTVi2xNtXR3Q2G5Hx1RVYVFoNsQED8+FPCgjy6m7ZQbxKP1sjCJt5rbtBg==",
"license": "SEE LICENSE IN LICENSE",
"dependencies": {
"bpmn-moddle": "^10.0.0",
"diagram-js": "^15.9.0",
"diagram-js-direct-editing": "^3.3.0",
"ids": "^3.0.0",
"inherits-browser": "^0.1.0",
"min-dash": "^5.0.0",
"min-dom": "^5.2.0",
"tiny-svg": "^4.1.4"
},
"engines": {
"node": "*"
}
},
"node_modules/bpmn-moddle": {
"version": "10.0.0",
"resolved": "https://registry.npmjs.org/bpmn-moddle/-/bpmn-moddle-10.0.0.tgz",
"integrity": "sha512-vXePD5jkatcILmM3zwJG/m6IIHIghTGB7WvgcdEraEw8E8VdJHrTgrvBUhbzqaXJpnsGQz15QS936xeBY6l9aA==",
"license": "MIT",
"dependencies": {
"min-dash": "^5.0.0",
"moddle": "^8.0.0",
"moddle-xml": "^12.0.0"
},
"engines": {
"node": ">= 20.12"
}
},
"node_modules/braces": {
"version": "3.0.3",
"resolved": "https://registry.npmjs.org/braces/-/braces-3.0.3.tgz",
@@ -3153,6 +3198,15 @@
"integrity": "sha512-IV3Ou0jSMzZrd3pZ48nLkT9DA7Ag1pnPzaiQhpW7c3RbcqqzvzzVu+L8gfqMp/8IM2MQtSiqaCxrrcfu8I8rMA==",
"license": "MIT"
},
"node_modules/clsx": {
"version": "2.1.1",
"resolved": "https://registry.npmjs.org/clsx/-/clsx-2.1.1.tgz",
"integrity": "sha512-eYm0QWBtUrBWZWG0d386OGAw16Z995PiOVo2B7bjWSbHedGl5e0ZWaq65kOGgUSNesEIDkB9ISbTg/JK9dhCZA==",
"license": "MIT",
"engines": {
"node": ">=6"
}
},
"node_modules/commander": {
"version": "4.1.1",
"resolved": "https://registry.npmjs.org/commander/-/commander-4.1.1.tgz",
@@ -3262,9 +3316,20 @@
"version": "3.2.3",
"resolved": "https://registry.npmjs.org/csstype/-/csstype-3.2.3.tgz",
"integrity": "sha512-z1HGKcYy2xA8AGQfwrn0PAy+PB7X/GSj3UVJW9qKyn43xWa+gl5nXmU4qqLMRzWVLFC8KusUX8T/0kCiOYpAIQ==",
"devOptional": true,
"license": "MIT"
},
"node_modules/d3-array": {
"version": "3.2.4",
"resolved": "https://registry.npmjs.org/d3-array/-/d3-array-3.2.4.tgz",
"integrity": "sha512-tdQAmyA18i4J7wprpYq8ClcxZy3SC31QMeByyCFyRt7BVHdREQZ5lpzoe5mFEYZUWe+oq8HBvk9JjpibyEV4Jg==",
"license": "ISC",
"dependencies": {
"internmap": "1 - 2"
},
"engines": {
"node": ">=12"
}
},
"node_modules/d3-color": {
"version": "3.1.0",
"resolved": "https://registry.npmjs.org/d3-color/-/d3-color-3.1.0.tgz",
@@ -3305,6 +3370,15 @@
"node": ">=12"
}
},
"node_modules/d3-format": {
"version": "3.1.2",
"resolved": "https://registry.npmjs.org/d3-format/-/d3-format-3.1.2.tgz",
"integrity": "sha512-AJDdYOdnyRDV5b6ArilzCPPwc1ejkHcoyFarqlPqT7zRYjhavcT3uSrqcMvsgh2CgoPbK3RCwyHaVyxYcP2Arg==",
"license": "ISC",
"engines": {
"node": ">=12"
}
},
"node_modules/d3-interpolate": {
"version": "3.0.1",
"resolved": "https://registry.npmjs.org/d3-interpolate/-/d3-interpolate-3.0.1.tgz",
@@ -3317,6 +3391,31 @@
"node": ">=12"
}
},
"node_modules/d3-path": {
"version": "3.1.0",
"resolved": "https://registry.npmjs.org/d3-path/-/d3-path-3.1.0.tgz",
"integrity": "sha512-p3KP5HCf/bvjBSSKuXid6Zqijx7wIfNW+J/maPs+iwR35at5JCbLUT0LzF1cnjbCHWhqzQTIN2Jpe8pRebIEFQ==",
"license": "ISC",
"engines": {
"node": ">=12"
}
},
"node_modules/d3-scale": {
"version": "4.0.2",
"resolved": "https://registry.npmjs.org/d3-scale/-/d3-scale-4.0.2.tgz",
"integrity": "sha512-GZW464g1SH7ag3Y7hXjf8RoUuAFIqklOAq3MRl4OaWabTFJY9PN/E1YklhXLh+OQ3fM9yS2nOkCoS+WLZ6kvxQ==",
"license": "ISC",
"dependencies": {
"d3-array": "2.10.0 - 3",
"d3-format": "1 - 3",
"d3-interpolate": "1.2.0 - 3",
"d3-time": "2.1.1 - 3",
"d3-time-format": "2 - 4"
},
"engines": {
"node": ">=12"
}
},
"node_modules/d3-selection": {
"version": "3.0.0",
"resolved": "https://registry.npmjs.org/d3-selection/-/d3-selection-3.0.0.tgz",
@@ -3326,6 +3425,42 @@
"node": ">=12"
}
},
"node_modules/d3-shape": {
"version": "3.2.0",
"resolved": "https://registry.npmjs.org/d3-shape/-/d3-shape-3.2.0.tgz",
"integrity": "sha512-SaLBuwGm3MOViRq2ABk3eLoxwZELpH6zhl3FbAoJ7Vm1gofKx6El1Ib5z23NUEhF9AsGl7y+dzLe5Cw2AArGTA==",
"license": "ISC",
"dependencies": {
"d3-path": "^3.1.0"
},
"engines": {
"node": ">=12"
}
},
"node_modules/d3-time": {
"version": "3.1.0",
"resolved": "https://registry.npmjs.org/d3-time/-/d3-time-3.1.0.tgz",
"integrity": "sha512-VqKjzBLejbSMT4IgbmVgDjpkYrNWUYJnbCGo874u7MMKIWsILRX+OpX/gTk8MqjpT1A/c6HY2dCA77ZN0lkQ2Q==",
"license": "ISC",
"dependencies": {
"d3-array": "2 - 3"
},
"engines": {
"node": ">=12"
}
},
"node_modules/d3-time-format": {
"version": "4.1.0",
"resolved": "https://registry.npmjs.org/d3-time-format/-/d3-time-format-4.1.0.tgz",
"integrity": "sha512-dJxPBlzC7NugB2PDLwo9Q8JiTR3M3e4/XANkreKSUxF8vvXKqm1Yfq4Q5dl8budlunRVlUUaDUgFt7eA8D6NLg==",
"license": "ISC",
"dependencies": {
"d3-time": "1 - 3"
},
"engines": {
"node": ">=12"
}
},
"node_modules/d3-timer": {
"version": "3.0.1",
"resolved": "https://registry.npmjs.org/d3-timer/-/d3-timer-3.0.1.tgz",
@@ -3409,6 +3544,12 @@
"dev": true,
"license": "MIT"
},
"node_modules/decimal.js-light": {
"version": "2.5.1",
"resolved": "https://registry.npmjs.org/decimal.js-light/-/decimal.js-light-2.5.1.tgz",
"integrity": "sha512-qIMFpTMZmny+MMIitAB6D7iVPEorVw6YQRWkvarTkT4tBeSLLiHzcwj6q0MmYSFCiVpiqPJTJEYIrpcPzVEIvg==",
"license": "MIT"
},
"node_modules/dequal": {
"version": "2.0.3",
"resolved": "https://registry.npmjs.org/dequal/-/dequal-2.0.3.tgz",
@@ -3429,6 +3570,51 @@
"node": ">=8"
}
},
"node_modules/diagram-js": {
"version": "15.9.1",
"resolved": "https://registry.npmjs.org/diagram-js/-/diagram-js-15.9.1.tgz",
"integrity": "sha512-2JsGmyeTo6o39beq2e/UkTfMopQSM27eXBUzbYQ+1m5VhEnQDkcjcrnRCjcObLMzzXSE/LSJyYhji90sqBFodQ==",
"license": "MIT",
"dependencies": {
"@bpmn-io/diagram-js-ui": "^0.2.3",
"clsx": "^2.1.1",
"didi": "^11.0.0",
"inherits-browser": "^0.1.0",
"min-dash": "^5.0.0",
"min-dom": "^5.2.0",
"object-refs": "^0.4.0",
"path-intersection": "^4.1.0",
"tiny-svg": "^4.1.4"
},
"engines": {
"node": "*"
}
},
"node_modules/diagram-js-direct-editing": {
"version": "3.3.0",
"resolved": "https://registry.npmjs.org/diagram-js-direct-editing/-/diagram-js-direct-editing-3.3.0.tgz",
"integrity": "sha512-EjXYb35J3qBU8lLz5U81hn7wNykVmF7U5DXZ7BvPok2IX7rmPz+ZyaI5AEMiqaC6lpSnHqPxFcPgKEiJcAiv5w==",
"license": "MIT",
"dependencies": {
"min-dash": "^5.0.0",
"min-dom": "^5.2.0"
},
"engines": {
"node": "*"
},
"peerDependencies": {
"diagram-js": "*"
}
},
"node_modules/didi": {
"version": "11.0.0",
"resolved": "https://registry.npmjs.org/didi/-/didi-11.0.0.tgz",
"integrity": "sha512-PzCfRzQttvFpVcYMbSF7h8EsWjeJpVjWH4qDhB5LkMi1ILvHq4Ob0vhM2wLFziPkbUBi+PAo7ODbe2sacR7nJQ==",
"license": "MIT",
"engines": {
"node": ">= 20.12"
}
},
"node_modules/didyoumean": {
"version": "1.2.2",
"resolved": "https://registry.npmjs.org/didyoumean/-/didyoumean-1.2.2.tgz",
@@ -3451,6 +3637,28 @@
"license": "MIT",
"peer": true
},
"node_modules/dom-helpers": {
"version": "5.2.1",
"resolved": "https://registry.npmjs.org/dom-helpers/-/dom-helpers-5.2.1.tgz",
"integrity": "sha512-nRCa7CK3VTrM2NmGkIy4cbK7IZlgBE/PYMn55rrXefr5xXDP0LdtfPnblFDoVdcAfslJ7or6iqAUnx0CCGIWQA==",
"license": "MIT",
"dependencies": {
"@babel/runtime": "^7.8.7",
"csstype": "^3.0.2"
}
},
"node_modules/domify": {
"version": "3.0.0",
"resolved": "https://registry.npmjs.org/domify/-/domify-3.0.0.tgz",
"integrity": "sha512-bs2yO68JDFOm6rKv8f0EnrM2cENduhRkpqOtt/s5l5JBA/eqGBZCzLPmdYoHtJ6utgLGgcBajFsEQbl12pT0lQ==",
"license": "MIT",
"engines": {
"node": ">=20"
},
"funding": {
"url": "https://github.com/sponsors/sindresorhus"
}
},
"node_modules/dompurify": {
"version": "3.3.1",
"resolved": "https://registry.npmjs.org/dompurify/-/dompurify-3.3.1.tgz",
@@ -3550,6 +3758,12 @@
"@types/estree": "^1.0.0"
}
},
"node_modules/eventemitter3": {
"version": "4.0.7",
"resolved": "https://registry.npmjs.org/eventemitter3/-/eventemitter3-4.0.7.tgz",
"integrity": "sha512-8guHBZCwKnFhYdHr2ysuRWErTwhoN2X8XELRlrRwpmfeY2jjuUN4taQMsULKUVo1K4DvZl+0pgfyoysHxvmvEw==",
"license": "MIT"
},
"node_modules/expect-type": {
"version": "1.3.0",
"resolved": "https://registry.npmjs.org/expect-type/-/expect-type-1.3.0.tgz",
@@ -3560,6 +3774,15 @@
"node": ">=12.0.0"
}
},
"node_modules/fast-equals": {
"version": "5.4.0",
"resolved": "https://registry.npmjs.org/fast-equals/-/fast-equals-5.4.0.tgz",
"integrity": "sha512-jt2DW/aNFNwke7AUd+Z+e6pz39KO5rzdbbFCg2sGafS4mk13MI7Z8O5z9cADNn5lhGODIgLwug6TZO2ctf7kcw==",
"license": "MIT",
"engines": {
"node": ">=6.0.0"
}
},
"node_modules/fast-glob": {
"version": "3.3.3",
"resolved": "https://registry.npmjs.org/fast-glob/-/fast-glob-3.3.3.tgz",
@@ -3705,6 +3928,12 @@
"node": ">= 0.4"
}
},
"node_modules/htm": {
"version": "3.1.1",
"resolved": "https://registry.npmjs.org/htm/-/htm-3.1.1.tgz",
"integrity": "sha512-983Vyg8NwUE7JkZ6NmOqpCZ+sh1bKv2iYTlUkzlWmA5JD2acKoxd4KVxbMmxX/85mtfdnDmTFoNKcg5DGAvxNQ==",
"license": "Apache-2.0"
},
"node_modules/html-encoding-sniffer": {
"version": "6.0.0",
"resolved": "https://registry.npmjs.org/html-encoding-sniffer/-/html-encoding-sniffer-6.0.0.tgz",
@@ -3760,6 +3989,15 @@
"node": ">= 14"
}
},
"node_modules/ids": {
"version": "3.0.1",
"resolved": "https://registry.npmjs.org/ids/-/ids-3.0.1.tgz",
"integrity": "sha512-mr0zAgpgA/hzCrHB0DnoTG6xZjNC3ABs4eaksXrpVtfaDatA2SVdDb1ZPLjmKjqzp4kexQRuHXwDWQILVK8FZQ==",
"license": "MIT",
"engines": {
"node": ">= 20.12"
}
},
"node_modules/immediate": {
"version": "3.0.6",
"resolved": "https://registry.npmjs.org/immediate/-/immediate-3.0.6.tgz",
@@ -3782,6 +4020,21 @@
"integrity": "sha512-k/vGaX4/Yla3WzyMCvTQOXYeIHvqOKtnqBduzTHpzpQZzAskKMhZ2K+EnBiSM9zGSoIFeMpXKxa4dYeZIQqewQ==",
"license": "ISC"
},
"node_modules/inherits-browser": {
"version": "0.1.0",
"resolved": "https://registry.npmjs.org/inherits-browser/-/inherits-browser-0.1.0.tgz",
"integrity": "sha512-CJHHvW3jQ6q7lzsXPpapLdMx5hDpSF3FSh45pwsj6bKxJJ8Nl8v43i5yXnr3BdfOimGHKyniewQtnAIp3vyJJw==",
"license": "ISC"
},
"node_modules/internmap": {
"version": "2.0.3",
"resolved": "https://registry.npmjs.org/internmap/-/internmap-2.0.3.tgz",
"integrity": "sha512-5Hh7Y1wQbvY5ooGgPbDaL5iYLAPzMTUrjMulskHLH6wnv/A+1q5rgEaiuqEjB+oxGXIVZs1FF+R/KPN3ZSQYYg==",
"license": "ISC",
"engines": {
"node": ">=12"
}
},
"node_modules/iobuffer": {
"version": "5.4.0",
"resolved": "https://registry.npmjs.org/iobuffer/-/iobuffer-5.4.0.tgz",
@@ -4009,6 +4262,12 @@
"dev": true,
"license": "MIT"
},
"node_modules/lodash": {
"version": "4.17.23",
"resolved": "https://registry.npmjs.org/lodash/-/lodash-4.17.23.tgz",
"integrity": "sha512-LgVTMpQtIopCi79SJeDiP0TfWi5CNEc/L/aRdTh3yIvmZXTnheWpKjSZhnvMl8iXbC1tFg9gdHHDMLoV7CnG+w==",
"license": "MIT"
},
"node_modules/loose-envify": {
"version": "1.4.0",
"resolved": "https://registry.npmjs.org/loose-envify/-/loose-envify-1.4.0.tgz",
@@ -4092,6 +4351,22 @@
"node": ">=8.6"
}
},
"node_modules/min-dash": {
"version": "5.0.0",
"resolved": "https://registry.npmjs.org/min-dash/-/min-dash-5.0.0.tgz",
"integrity": "sha512-EGuoBnVL7/Fnv2sqakpX5WGmZehZ3YMmLayT7sM8E9DRU74kkeyMg4Rik1lsOkR2GbFNeBca4/L+UfU6gF0Edw==",
"license": "MIT"
},
"node_modules/min-dom": {
"version": "5.3.0",
"resolved": "https://registry.npmjs.org/min-dom/-/min-dom-5.3.0.tgz",
"integrity": "sha512-0w5FEBgPAyHhmFojW3zxd7we3D+m5XYS3E/06OyvxmbHJoiQVa4Nagj6RWvoAKYRw5xth6cP5TMePc5cR1M9hA==",
"license": "MIT",
"dependencies": {
"domify": "^3.0.0",
"min-dash": "^5.0.0"
}
},
"node_modules/min-indent": {
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/min-indent/-/min-indent-1.0.1.tgz",
@@ -4102,6 +4377,31 @@
"node": ">=4"
}
},
"node_modules/moddle": {
"version": "8.1.0",
"resolved": "https://registry.npmjs.org/moddle/-/moddle-8.1.0.tgz",
"integrity": "sha512-dBddc1CNuZHgro8nQWwfPZ2BkyLWdnxoNpPu9d+XKPN96DAiiBOeBw527ft++ebDuFez5PMdaR3pgUgoOaUGrA==",
"license": "MIT",
"dependencies": {
"min-dash": "^5.0.0"
}
},
"node_modules/moddle-xml": {
"version": "12.0.0",
"resolved": "https://registry.npmjs.org/moddle-xml/-/moddle-xml-12.0.0.tgz",
"integrity": "sha512-NJc2+sCe4tvuGlaUBcoZcYf6j9f+z+qxHOyGm/LB3ZrlJXVPPHoBTg/KXgDRCufdBJhJ3AheFs3QU/abABNzRg==",
"license": "MIT",
"dependencies": {
"min-dash": "^5.0.0",
"saxen": "^11.0.2"
},
"engines": {
"node": ">= 18"
},
"peerDependencies": {
"moddle": ">= 6.2.0"
}
},
"node_modules/ms": {
"version": "2.1.3",
"resolved": "https://registry.npmjs.org/ms/-/ms-2.1.3.tgz",
@@ -4240,7 +4540,6 @@
"version": "4.1.1",
"resolved": "https://registry.npmjs.org/object-assign/-/object-assign-4.1.1.tgz",
"integrity": "sha512-rJgTQnkUnH1sFw8yT6VSU3zD3sWmu6sZhIseY8VX+GRu3P6F7Fu+JNDoXfklElbLJSnc3FUQHVe4cU5hj+BcUg==",
"dev": true,
"license": "MIT",
"engines": {
"node": ">=0.10.0"
@@ -4256,6 +4555,15 @@
"node": ">= 6"
}
},
"node_modules/object-refs": {
"version": "0.4.0",
"resolved": "https://registry.npmjs.org/object-refs/-/object-refs-0.4.0.tgz",
"integrity": "sha512-6kJqKWryKZmtte6QYvouas0/EIJKPI1/MMIuRsiBlNuhIMfqYTggzX2F1AJ2+cDs288xyi9GL7FyasHINR98BQ==",
"license": "MIT",
"engines": {
"node": "*"
}
},
"node_modules/obug": {
"version": "2.1.1",
"resolved": "https://registry.npmjs.org/obug/-/obug-2.1.1.tgz",
@@ -4286,6 +4594,15 @@
"url": "https://github.com/inikulin/parse5?sponsor=1"
}
},
"node_modules/path-intersection": {
"version": "4.1.0",
"resolved": "https://registry.npmjs.org/path-intersection/-/path-intersection-4.1.0.tgz",
"integrity": "sha512-urUP6WvhnxbHPdHYl6L7Yrc6+1ny6uOFKPCzPxTSUSYGHG0o94RmI7SvMMaScNAM5RtTf08bg4skc6/kjfne3A==",
"license": "MIT",
"engines": {
"node": ">= 14.20"
}
},
"node_modules/path-parse": {
"version": "1.0.7",
"resolved": "https://registry.npmjs.org/path-parse/-/path-parse-1.0.7.tgz",
@@ -4555,6 +4872,16 @@
"dev": true,
"license": "MIT"
},
"node_modules/preact": {
"version": "10.28.4",
"resolved": "https://registry.npmjs.org/preact/-/preact-10.28.4.tgz",
"integrity": "sha512-uKFfOHWuSNpRFVTnljsCluEFq57OKT+0QdOiQo8XWnQ/pSvg7OpX5eNOejELXJMWy+BwM2nobz0FkvzmnpCNsQ==",
"license": "MIT",
"funding": {
"type": "opencollective",
"url": "https://opencollective.com/preact"
}
},
"node_modules/pretty-format": {
"version": "27.5.1",
"resolved": "https://registry.npmjs.org/pretty-format/-/pretty-format-27.5.1.tgz",
@@ -4577,6 +4904,23 @@
"integrity": "sha512-3ouUOpQhtgrbOa17J7+uxOTpITYWaGP7/AhoR3+A+/1e9skrzelGi/dXzEYyvbxubEF6Wn2ypscTKiKJFFn1ag==",
"license": "MIT"
},
"node_modules/prop-types": {
"version": "15.8.1",
"resolved": "https://registry.npmjs.org/prop-types/-/prop-types-15.8.1.tgz",
"integrity": "sha512-oj87CgZICdulUohogVAR7AjlC0327U4el4L6eAvOqCeudMDVU0NThNaV+b9Df4dXgSP1gXMTnPdhfe/2qDH5cg==",
"license": "MIT",
"dependencies": {
"loose-envify": "^1.4.0",
"object-assign": "^4.1.1",
"react-is": "^16.13.1"
}
},
"node_modules/prop-types/node_modules/react-is": {
"version": "16.13.1",
"resolved": "https://registry.npmjs.org/react-is/-/react-is-16.13.1.tgz",
"integrity": "sha512-24e6ynE2H+OKt4kqsOvNd8kBpV65zoxbA4BVsEOB3ARVWQki/DHzaUoC5KuON/BiccDaCCTZBuOcfZs70kR8bQ==",
"license": "MIT"
},
"node_modules/punycode": {
"version": "2.3.1",
"resolved": "https://registry.npmjs.org/punycode/-/punycode-2.3.1.tgz",
@@ -4661,6 +5005,37 @@
"node": ">=0.10.0"
}
},
"node_modules/react-smooth": {
"version": "4.0.4",
"resolved": "https://registry.npmjs.org/react-smooth/-/react-smooth-4.0.4.tgz",
"integrity": "sha512-gnGKTpYwqL0Iii09gHobNolvX4Kiq4PKx6eWBCYYix+8cdw+cGo3do906l1NBPKkSWx1DghC1dlWG9L2uGd61Q==",
"license": "MIT",
"dependencies": {
"fast-equals": "^5.0.1",
"prop-types": "^15.8.1",
"react-transition-group": "^4.4.5"
},
"peerDependencies": {
"react": "^16.8.0 || ^17.0.0 || ^18.0.0 || ^19.0.0",
"react-dom": "^16.8.0 || ^17.0.0 || ^18.0.0 || ^19.0.0"
}
},
"node_modules/react-transition-group": {
"version": "4.4.5",
"resolved": "https://registry.npmjs.org/react-transition-group/-/react-transition-group-4.4.5.tgz",
"integrity": "sha512-pZcd1MCJoiKiBR2NRxeCRg13uCXbydPnmB4EOeRrY7480qNWO8IIgQG6zlDkm6uRMsURXPuKq0GWtiM59a5Q6g==",
"license": "BSD-3-Clause",
"dependencies": {
"@babel/runtime": "^7.5.5",
"dom-helpers": "^5.0.1",
"loose-envify": "^1.4.0",
"prop-types": "^15.6.2"
},
"peerDependencies": {
"react": ">=16.6.0",
"react-dom": ">=16.6.0"
}
},
"node_modules/reactflow": {
"version": "11.11.4",
"resolved": "https://registry.npmjs.org/reactflow/-/reactflow-11.11.4.tgz",
@@ -4717,6 +5092,44 @@
"node": ">=8.10.0"
}
},
"node_modules/recharts": {
"version": "2.15.4",
"resolved": "https://registry.npmjs.org/recharts/-/recharts-2.15.4.tgz",
"integrity": "sha512-UT/q6fwS3c1dHbXv2uFgYJ9BMFHu3fwnd7AYZaEQhXuYQ4hgsxLvsUXzGdKeZrW5xopzDCvuA2N41WJ88I7zIw==",
"license": "MIT",
"dependencies": {
"clsx": "^2.0.0",
"eventemitter3": "^4.0.1",
"lodash": "^4.17.21",
"react-is": "^18.3.1",
"react-smooth": "^4.0.4",
"recharts-scale": "^0.4.4",
"tiny-invariant": "^1.3.1",
"victory-vendor": "^36.6.8"
},
"engines": {
"node": ">=14"
},
"peerDependencies": {
"react": "^16.0.0 || ^17.0.0 || ^18.0.0 || ^19.0.0",
"react-dom": "^16.0.0 || ^17.0.0 || ^18.0.0 || ^19.0.0"
}
},
"node_modules/recharts-scale": {
"version": "0.4.5",
"resolved": "https://registry.npmjs.org/recharts-scale/-/recharts-scale-0.4.5.tgz",
"integrity": "sha512-kivNFO+0OcUNu7jQquLXAxz1FIwZj8nrj+YkOKc5694NbjCvcT6aSZiIzNzd2Kul4o4rTto8QVR9lMNtxD4G1w==",
"license": "MIT",
"dependencies": {
"decimal.js-light": "^2.4.1"
}
},
"node_modules/recharts/node_modules/react-is": {
"version": "18.3.1",
"resolved": "https://registry.npmjs.org/react-is/-/react-is-18.3.1.tgz",
"integrity": "sha512-/LLMVyas0ljjAtoYiPqYiL8VWXzUUdThrmU5+n20DZv+a+ClRoevUzw5JxU+Ieh5/c87ytoTBV9G1FiKfNJdmg==",
"license": "MIT"
},
"node_modules/redent": {
"version": "3.0.0",
"resolved": "https://registry.npmjs.org/redent/-/redent-3.0.0.tgz",
@@ -4865,6 +5278,15 @@
"integrity": "sha512-Gd2UZBJDkXlY7GbJxfsE8/nvKkUEU1G38c1siN6QP6a9PT9MmHB8GnpscSmMJSoF8LOIrt8ud/wPtojys4G6+g==",
"license": "MIT"
},
"node_modules/saxen": {
"version": "11.0.2",
"resolved": "https://registry.npmjs.org/saxen/-/saxen-11.0.2.tgz",
"integrity": "sha512-WDb4gqac8uiJzOdOdVpr9NWh9NrJMm7Brn5GX2Poj+mjE/QTXqYQENr8T/mom54dDDgbd3QjwTg23TRHYiWXRA==",
"license": "MIT",
"engines": {
"node": ">= 20.12"
}
},
"node_modules/saxes": {
"version": "6.0.0",
"resolved": "https://registry.npmjs.org/saxes/-/saxes-6.0.0.tgz",
@@ -5160,6 +5582,21 @@
"node": ">=0.8"
}
},
"node_modules/tiny-invariant": {
"version": "1.3.3",
"resolved": "https://registry.npmjs.org/tiny-invariant/-/tiny-invariant-1.3.3.tgz",
"integrity": "sha512-+FbBPE1o9QAYvviau/qC5SE3caw21q3xkvWKBtja5vgqOWIHHJ3ioaq1VPfn/Szqctz2bU/oYeKd9/z5BL+PVg==",
"license": "MIT"
},
"node_modules/tiny-svg": {
"version": "4.1.4",
"resolved": "https://registry.npmjs.org/tiny-svg/-/tiny-svg-4.1.4.tgz",
"integrity": "sha512-cBaEACCbouYrQc9RG+eTXnPYosX1Ijqty/I6DdXovwDd89Pwu4jcmpOR7BuFEF9YCcd7/AWwasE0207WMK7hdw==",
"license": "MIT",
"engines": {
"node": ">= 20"
}
},
"node_modules/tinybench": {
"version": "2.9.0",
"resolved": "https://registry.npmjs.org/tinybench/-/tinybench-2.9.0.tgz",
@@ -5407,6 +5844,28 @@
"uuid": "dist-node/bin/uuid"
}
},
"node_modules/victory-vendor": {
"version": "36.9.2",
"resolved": "https://registry.npmjs.org/victory-vendor/-/victory-vendor-36.9.2.tgz",
"integrity": "sha512-PnpQQMuxlwYdocC8fIJqVXvkeViHYzotI+NJrCuav0ZYFoq912ZHBk3mCeuj+5/VpodOjPe1z0Fk2ihgzlXqjQ==",
"license": "MIT AND ISC",
"dependencies": {
"@types/d3-array": "^3.0.3",
"@types/d3-ease": "^3.0.0",
"@types/d3-interpolate": "^3.0.1",
"@types/d3-scale": "^4.0.2",
"@types/d3-shape": "^3.1.0",
"@types/d3-time": "^3.0.0",
"@types/d3-timer": "^3.0.0",
"d3-array": "^3.1.6",
"d3-ease": "^3.0.1",
"d3-interpolate": "^3.0.1",
"d3-scale": "^4.0.2",
"d3-shape": "^3.1.0",
"d3-time": "^3.0.0",
"d3-timer": "^3.0.1"
}
},
"node_modules/vite": {
"version": "7.3.1",
"resolved": "https://registry.npmjs.org/vite/-/vite-7.3.1.tgz",

File diff suppressed because one or more lines are too long

View File

@@ -0,0 +1,114 @@
# Chunk-Browser
## Uebersicht
Der Chunk-Browser ermoeglicht das sequenzielle Durchblaettern aller Chunks in einer Qdrant-Collection. Er ist als Tab "Chunk-Browser" auf der RAG-Seite (`/ai/rag`) verfuegbar.
**URL:** `https://macmini:3002/ai/rag` → Tab "Chunk-Browser"
---
## Funktionen
### Collection-Auswahl
Dropdown mit allen verfuegbaren Compliance-Collections:
- `bp_compliance_gesetze`
- `bp_compliance_ce`
- `bp_compliance_datenschutz`
- `bp_dsfa_corpus`
- `bp_compliance_recht`
- `bp_legal_templates`
- `bp_compliance_gdpr`
- `bp_compliance_schulrecht`
- `bp_dsfa_templates`
- `bp_dsfa_risks`
### Seitenweise Navigation
- 20 Chunks pro Seite
- Zurueck/Weiter-Buttons
- Seitennummer und Chunk-Zaehler
- Cursor-basierte Pagination via Qdrant Scroll API
### Textsuche
- Filtert Chunks auf der aktuell geladenen Seite
- Treffer werden gelb hervorgehoben
- Suche ueber den Chunk-Text (payload.text, payload.content, payload.chunk_text)
### Chunk-Details
- Klick auf einen Chunk klappt alle Metadaten aus
- Zeigt: regulation_code, article, language, source, licence, etc.
- Chunks haben eine fortlaufende Nummer (#1, #2, ...)
### Integration mit Regulierungen-Tab
Der Button "In Chunks suchen" bei jeder Regulierung wechselt zum Chunk-Browser mit:
- Vorauswahl der richtigen Collection
- Vorausgefuelltem Suchbegriff (Regulierungsname)
---
## API
### Scroll-Endpoint (API Proxy)
```
GET /api/legal-corpus?action=scroll&collection=bp_compliance_ce&limit=20&offset={cursor}
```
**Parameter:**
| Parameter | Typ | Beschreibung |
|-----------|-----|--------------|
| `collection` | string | Qdrant Collection Name |
| `limit` | number | Chunks pro Seite (max 100) |
| `offset` | string | Cursor fuer naechste Seite (optional) |
| `text_search` | string | Textsuche-Filter (optional) |
**Response:**
```json
{
"chunks": [
{
"id": "uuid",
"text": "...",
"regulation_code": "GDPR",
"article": "Art. 5",
"language": "de"
}
],
"next_offset": "uuid-or-null",
"total_in_page": 20
}
```
### Collection-Count-Endpoint
```
GET /api/legal-corpus?action=collection-count&collection=bp_compliance_ce
```
**Response:**
```json
{
"count": 12345
}
```
---
## Technische Details
- Der API-Proxy spricht direkt mit Qdrant (Port 6333) via dessen `POST /collections/{name}/points/scroll` Endpoint
- Kein Embedding oder rag-service erforderlich
- Textsuche ist client-seitig (kein Embedding noetig)
- Pagination ist cursor-basiert (Qdrant `next_page_offset`)
---
## Weitere Features auf der RAG-Seite
### Originalquelle-Links
Jede Regulierung in der Tabelle hat einen "Originalquelle" Link zum offiziellen Dokument (EUR-Lex, gesetze-im-internet.de, etc.). Definiert in `REGULATION_SOURCES` (88 Eintraege).
### Low-Chunk-Warnung
Regulierungen mit weniger als 10 Chunks aber einem erwarteten Wert >= 10 werden mit einem Amber-Warnsymbol markiert. Dies hilft, fehlgeschlagene oder unvollstaendige Ingestions zu erkennen.

View File

@@ -0,0 +1,373 @@
# OCR Pipeline - Schrittweise Seitenrekonstruktion
**Version:** 1.0.0
**Status:** In Entwicklung
**URL:** https://macmini:3002/ai/ocr-pipeline
## Uebersicht
Die OCR Pipeline zerlegt den OCR-Prozess in **8 einzelne Schritte**, um eingescannte Vokabelseiten Wort fuer Wort zu rekonstruieren. Jeder Schritt kann individuell geprueft, korrigiert und mit Ground-Truth-Daten versehen werden.
**Ziel:** 10 Vokabelseiten fehlerfrei rekonstruieren.
### Pipeline-Schritte
| Schritt | Name | Beschreibung | Status |
|---------|------|--------------|--------|
| 1 | Begradigung (Deskew) | Scan begradigen (Hough Lines + Word Alignment) | Implementiert |
| 2 | Entzerrung (Dewarp) | Buchwoelbung entzerren (Vertikalkanten-Analyse) | Implementiert |
| 3 | Spaltenerkennung | Unsichtbare Spalten finden (Projektionsprofile) | Implementiert |
| 4 | Zeilenerkennung | Horizontale Zeilen + Kopf-/Fusszeilen-Klassifikation | Implementiert |
| 5 | Worterkennung | Grid aus Spalten x Zeilen, OCR pro Zelle | Implementiert |
| 6 | Koordinatenzuweisung | Exakte Positionen innerhalb Zellen | Geplant |
| 7 | Seitenrekonstruktion | Seite nachbauen aus Koordinaten | Geplant |
| 8 | Ground Truth Validierung | Gesamtpruefung aller Schritte | Geplant |
---
## Architektur
```
Admin-Lehrer (Next.js) klausur-service (FastAPI :8086)
┌────────────────────┐ ┌─────────────────────────────┐
│ /ai/ocr-pipeline │ │ /api/v1/ocr-pipeline/ │
│ │ REST │ │
│ PipelineStepper │◄────────►│ Sessions CRUD │
│ StepDeskew │ │ Image Serving │
│ StepDewarp │ │ Deskew/Dewarp/Columns/Rows │
│ StepColumnDetection│ │ Word Recognition │
│ StepRowDetection │ │ Ground Truth │
│ StepWordRecognition│ │ Overlay Images │
└────────────────────┘ └─────────────────────────────┘
┌─────────────────────┐
│ PostgreSQL │
│ ocr_pipeline_sessions│
│ (Images + JSONB) │
└─────────────────────┘
```
### Dateistruktur
```
klausur-service/backend/
├── ocr_pipeline_api.py # FastAPI Router (alle Endpoints)
├── ocr_pipeline_session_store.py # PostgreSQL Persistence
├── cv_vocab_pipeline.py # Computer Vision Algorithmen
└── migrations/
├── 002_ocr_pipeline_sessions.sql # Basis-Schema
├── 003_add_row_result.sql # Row-Result Spalte
└── 004_add_word_result.sql # Word-Result Spalte
admin-lehrer/
├── app/(admin)/ai/ocr-pipeline/
│ ├── page.tsx # Haupt-Page mit Session-Management
│ └── types.ts # TypeScript Interfaces
└── components/ocr-pipeline/
├── PipelineStepper.tsx # Fortschritts-Stepper
├── StepDeskew.tsx # Schritt 1
├── StepDewarp.tsx # Schritt 2
├── StepColumnDetection.tsx # Schritt 3
├── StepRowDetection.tsx # Schritt 4
├── StepWordRecognition.tsx # Schritt 5
├── StepCoordinates.tsx # Schritt 6 (Platzhalter)
├── StepReconstruction.tsx # Schritt 7 (Platzhalter)
└── StepGroundTruth.tsx # Schritt 8 (Platzhalter)
```
---
## API-Referenz
Alle Endpoints unter `/api/v1/ocr-pipeline/`.
### Sessions
| Methode | Pfad | Beschreibung |
|---------|------|--------------|
| `POST` | `/sessions` | Neue Session erstellen (Bild hochladen) |
| `GET` | `/sessions` | Alle Sessions auflisten |
| `GET` | `/sessions/{id}` | Session-Info mit allen Step-Results |
| `PUT` | `/sessions/{id}` | Session umbenennen |
| `DELETE` | `/sessions/{id}` | Session loeschen |
### Bilder
| Methode | Pfad | Beschreibung |
|---------|------|--------------|
| `GET` | `/sessions/{id}/image/original` | Originalbild |
| `GET` | `/sessions/{id}/image/deskewed` | Begradigtes Bild |
| `GET` | `/sessions/{id}/image/dewarped` | Entzerrtes Bild |
| `GET` | `/sessions/{id}/image/binarized` | Binarisiertes Bild |
| `GET` | `/sessions/{id}/image/columns-overlay` | Spalten-Overlay |
| `GET` | `/sessions/{id}/image/rows-overlay` | Zeilen-Overlay |
| `GET` | `/sessions/{id}/image/words-overlay` | Wort-Grid-Overlay |
### Schritt 1: Begradigung
| Methode | Pfad | Beschreibung |
|---------|------|--------------|
| `POST` | `/sessions/{id}/deskew` | Automatische Begradigung |
| `POST` | `/sessions/{id}/deskew/manual` | Manuelle Winkelkorrektur |
| `POST` | `/sessions/{id}/ground-truth/deskew` | Ground Truth speichern |
### Schritt 2: Entzerrung
| Methode | Pfad | Beschreibung |
|---------|------|--------------|
| `POST` | `/sessions/{id}/dewarp` | Automatische Entzerrung |
| `POST` | `/sessions/{id}/dewarp/manual` | Manueller Scherbungswinkel |
| `POST` | `/sessions/{id}/ground-truth/dewarp` | Ground Truth speichern |
### Schritt 3: Spalten
| Methode | Pfad | Beschreibung |
|---------|------|--------------|
| `POST` | `/sessions/{id}/columns` | Automatische Spaltenerkennung |
| `POST` | `/sessions/{id}/columns/manual` | Manuelle Spalten-Definition |
| `POST` | `/sessions/{id}/ground-truth/columns` | Ground Truth speichern |
### Schritt 4: Zeilen
| Methode | Pfad | Beschreibung |
|---------|------|--------------|
| `POST` | `/sessions/{id}/rows` | Automatische Zeilenerkennung |
| `POST` | `/sessions/{id}/rows/manual` | Manuelle Zeilen-Definition |
| `POST` | `/sessions/{id}/ground-truth/rows` | Ground Truth speichern |
| `GET` | `/sessions/{id}/ground-truth/rows` | Ground Truth abrufen |
### Schritt 5: Worterkennung
| Methode | Pfad | Beschreibung |
|---------|------|--------------|
| `POST` | `/sessions/{id}/words` | Wort-Grid aus Spalten x Zeilen erstellen |
| `POST` | `/sessions/{id}/ground-truth/words` | Ground Truth speichern |
| `GET` | `/sessions/{id}/ground-truth/words` | Ground Truth abrufen |
---
## Schritt 5: Worterkennung (Detail)
### Algorithmus: `build_word_grid()`
Schritt 5 nutzt die Ergebnisse von Schritt 3 (Spalten) und Schritt 4 (Zeilen), um ein Grid zu erstellen und jede Zelle per OCR auszulesen.
```
Spalten (Step 3): column_en | column_de | column_example
───────────┼─────────────┼────────────────
Zeilen (Step 4): R0 │ hello │ hallo │ Hello, World!
R1 │ world │ Welt │ The whole world
R2 │ book │ Buch │ Read a book
───────────┼─────────────┼────────────────
```
**Ablauf:**
1. **Filterung**: Nur `content`-Zeilen (kein Header/Footer) und relevante Spalten (`column_en`, `column_de`, `column_example`)
2. **Zell-Bildung**: Pro content-Zeile x pro relevante Spalte eine `PageRegion` berechnen
3. **OCR**: `ocr_region()` mit PSM 7 (Single Line) pro Zelle aufrufen
4. **Sprache**: `eng` fuer EN-Spalte, `deu` fuer DE-Spalte, `eng+deu` fuer Beispiele
5. **Gruppierung**: Zellen zu Vokabel-Eintraegen zusammenfuehren
### Response-Format
```json
{
"entries": [
{
"row_index": 0,
"english": "hello",
"german": "hallo",
"example": "Hello, how are you?",
"confidence": 85.3,
"bbox": {"x": 5.2, "y": 12.1, "w": 90.0, "h": 2.8},
"bbox_en": {"x": 5.2, "y": 12.1, "w": 30.0, "h": 2.8},
"bbox_de": {"x": 35.5, "y": 12.1, "w": 25.0, "h": 2.8},
"bbox_ex": {"x": 61.0, "y": 12.1, "w": 34.2, "h": 2.8}
}
],
"entry_count": 25,
"image_width": 2480,
"image_height": 3508,
"duration_seconds": 3.2,
"summary": {
"total_entries": 25,
"with_english": 24,
"with_german": 22,
"low_confidence": 3
}
}
```
!!! info "Bounding Boxes in Prozent"
Alle `bbox`-Werte sind Prozent (0-100) relativ zur Bildgroesse.
Das erleichtert die Darstellung im Frontend unabhaengig von der Bildaufloesung.
### Frontend: StepWordRecognition
Die Komponente bietet zwei Modi:
**Uebersicht-Modus:**
- Zwei Bilder nebeneinander: Grid-Overlay vs. sauberes Bild
- Tabelle aller erkannten Eintraege mit Konfidenz-Werten
- Klick auf Eintrag wechselt zum Labeling-Modus
**Labeling-Modus (Step-Through):**
- Links (2/3): Bild mit hervorgehobenem aktiven Eintrag (gelber Rahmen)
- Rechts (1/3): Zell-Ausschnitte + editierbare Felder (English, Deutsch, Example)
- Tastaturkuerzel:
- `Enter` = Bestaetigen und weiter
- `Ctrl+Pfeil runter` = Ueberspringen
- `Ctrl+Pfeil hoch` = Zurueck
**Feedback-Loop:**
- "Zeilen korrigieren" springt zurueck zu Schritt 4
- Nach Korrektur der Zeilen kann Schritt 5 erneut ausgefuehrt werden
---
## Datenbank-Schema
```sql
CREATE TABLE ocr_pipeline_sessions (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
name VARCHAR(255),
filename VARCHAR(255),
status VARCHAR(50) DEFAULT 'active',
current_step INT DEFAULT 1,
-- Bilder (BYTEA)
original_png BYTEA,
deskewed_png BYTEA,
binarized_png BYTEA,
dewarped_png BYTEA,
-- Step-Results (JSONB)
deskew_result JSONB,
dewarp_result JSONB,
column_result JSONB,
row_result JSONB,
word_result JSONB,
-- Ground Truth + Meta
ground_truth JSONB,
auto_shear_degrees REAL,
created_at TIMESTAMP DEFAULT NOW(),
updated_at TIMESTAMP DEFAULT NOW()
);
```
### Migrationen
| Datei | Beschreibung |
|-------|--------------|
| `002_ocr_pipeline_sessions.sql` | Basis-Schema (Steps 1-3) |
| `003_add_row_result.sql` | `row_result JSONB` fuer Step 4 |
| `004_add_word_result.sql` | `word_result JSONB` fuer Step 5 |
---
## TypeScript Interfaces
Die wichtigsten Typen in `types.ts`:
```typescript
interface WordEntry {
row_index: number
english: string
german: string
example: string
confidence: number
bbox: WordBbox // Gesamte Zeile
bbox_en: WordBbox | null // EN-Zelle
bbox_de: WordBbox | null // DE-Zelle
bbox_ex: WordBbox | null // Example-Zelle
status?: 'pending' | 'confirmed' | 'edited' | 'skipped'
}
interface WordResult {
entries: WordEntry[]
entry_count: number
image_width: number
image_height: number
duration_seconds: number
summary: {
total_entries: number
with_english: number
with_german: number
low_confidence: number
}
}
```
---
## Ground Truth System
Jeder Schritt kann mit Ground-Truth-Feedback versehen werden:
```json
{
"is_correct": false,
"corrected_entries": [...],
"notes": "Zeile 5 falsch erkannt",
"saved_at": "2026-02-28T10:30:00"
}
```
Ground-Truth-Daten werden in der `ground_truth` JSONB-Spalte gespeichert, gruppiert nach Schritt:
```json
{
"deskew": { "is_correct": true, ... },
"dewarp": { "is_correct": true, ... },
"columns": { "is_correct": false, ... },
"rows": { "is_correct": true, ... },
"words": { "is_correct": false, ... }
}
```
---
## Deployment
```bash
# 1. Git push
git push origin main && git push gitea main
# 2. Mac Mini pull + build
ssh macmini "cd /Users/benjaminadmin/Projekte/breakpilot-lehrer && git pull --no-rebase origin main"
# klausur-service (Backend)
ssh macmini "cd /Users/benjaminadmin/Projekte/breakpilot-lehrer && \
/usr/local/bin/docker compose build --no-cache klausur-service && \
/usr/local/bin/docker compose up -d klausur-service"
# admin-lehrer (Frontend)
ssh macmini "cd /Users/benjaminadmin/Projekte/breakpilot-lehrer && \
/usr/local/bin/docker compose build --no-cache admin-lehrer && \
/usr/local/bin/docker compose up -d admin-lehrer"
# 3. Migration ausfuehren
ssh macmini "/usr/local/bin/docker exec bp-lehrer-klausur-service \
python -c \"import asyncio; from ocr_pipeline_session_store import *; asyncio.run(init_ocr_pipeline_tables())\""
# 4. Testen unter:
# https://macmini:3002/ai/ocr-pipeline
```
---
## Aenderungshistorie
| Datum | Version | Aenderung |
|-------|---------|----------|
| 2026-02-28 | 1.0.0 | Schritt 5 (Worterkennung) implementiert |
| 2026-02-22 | 0.4.0 | Schritt 4 (Zeilenerkennung) implementiert |
| 2026-02-20 | 0.3.0 | Schritt 3 (Spaltenerkennung) mit Typ-Klassifikation |
| 2026-02-15 | 0.2.0 | Schritt 2 (Entzerrung/Dewarp) |
| 2026-02-12 | 0.1.0 | Schritt 1 (Begradigung/Deskew) + Session-Management |

View File

@@ -8,24 +8,15 @@ RUN npm install
COPY frontend/ ./
RUN npm run build
# Production stage
FROM python:3.11-slim
# Production stage — uses pre-built base with Tesseract + Python deps.
# Base image contains: python:3.11-slim + tesseract-ocr + all pip packages.
# Rebuild base only when requirements.txt or system deps change:
# docker build -f klausur-service/Dockerfile.base -t klausur-base:latest klausur-service/
FROM klausur-base:latest
WORKDIR /app
# Install system dependencies (incl. Tesseract OCR for bounding-box extraction)
RUN apt-get update && apt-get install -y --no-install-recommends \
curl \
tesseract-ocr \
tesseract-ocr-deu \
tesseract-ocr-eng \
&& rm -rf /var/lib/apt/lists/*
# Install Python dependencies
COPY backend/requirements.txt ./
RUN pip install --no-cache-dir -r requirements.txt
# Copy backend code
# Copy backend code (this is the only layer that changes on code edits)
COPY backend/ ./
# Copy built frontend to the expected path

View File

@@ -0,0 +1,26 @@
# Base image with system dependencies + Python packages.
# These change rarely — build once, reuse on every --no-cache.
#
# Rebuild manually when requirements.txt or system deps change:
# docker build -f klausur-service/Dockerfile.base -t klausur-base:latest klausur-service/
#
FROM python:3.11-slim
WORKDIR /app
# System dependencies (Tesseract OCR, curl for healthcheck)
RUN apt-get update && apt-get install -y --no-install-recommends \
curl \
tesseract-ocr \
tesseract-ocr-deu \
tesseract-ocr-eng \
libgl1 \
libglib2.0-0 \
&& rm -rf /var/lib/apt/lists/*
# Python dependencies
COPY backend/requirements.txt ./
RUN pip install --no-cache-dir -r requirements.txt
# Clean up pip cache
RUN rm -rf /root/.cache/pip

File diff suppressed because it is too large Load Diff

File diff suppressed because one or more lines are too long

View File

@@ -42,6 +42,8 @@ try:
except ImportError:
trocr_router = None
from vocab_worksheet_api import router as vocab_router, set_db_pool as set_vocab_db_pool, _init_vocab_table, _load_all_sessions, DATABASE_URL as VOCAB_DATABASE_URL
from ocr_pipeline_api import router as ocr_pipeline_router
from ocr_pipeline_session_store import init_ocr_pipeline_tables
try:
from dsfa_rag_api import router as dsfa_rag_router, set_db_pool as set_dsfa_db_pool
from dsfa_corpus_ingestion import DSFAQdrantService, DATABASE_URL as DSFA_DATABASE_URL
@@ -75,6 +77,13 @@ async def lifespan(app: FastAPI):
except Exception as e:
print(f"Warning: Vocab sessions database initialization failed: {e}")
# Initialize OCR Pipeline session tables
try:
await init_ocr_pipeline_tables()
print("OCR Pipeline session tables initialized")
except Exception as e:
print(f"Warning: OCR Pipeline tables initialization failed: {e}")
# Initialize database pool for DSFA RAG
dsfa_db_pool = None
if DSFA_DATABASE_URL and set_dsfa_db_pool:
@@ -150,6 +159,7 @@ app.include_router(mail_router) # Unified Inbox Mail
if trocr_router:
app.include_router(trocr_router) # TrOCR Handwriting OCR
app.include_router(vocab_router) # Vocabulary Worksheet Generator
app.include_router(ocr_pipeline_router) # OCR Pipeline (step-by-step)
if dsfa_rag_router:
app.include_router(dsfa_rag_router) # DSFA RAG Corpus Search

View File

@@ -0,0 +1,28 @@
-- OCR Pipeline Sessions - Persistent session storage
-- Applied automatically by ocr_pipeline_session_store.init_ocr_pipeline_tables()
CREATE TABLE IF NOT EXISTS ocr_pipeline_sessions (
id UUID PRIMARY KEY,
name VARCHAR(255) NOT NULL,
filename VARCHAR(255),
status VARCHAR(50) DEFAULT 'active',
current_step INT DEFAULT 1,
original_png BYTEA,
deskewed_png BYTEA,
binarized_png BYTEA,
dewarped_png BYTEA,
deskew_result JSONB,
dewarp_result JSONB,
column_result JSONB,
ground_truth JSONB DEFAULT '{}',
auto_shear_degrees FLOAT,
created_at TIMESTAMP DEFAULT NOW(),
updated_at TIMESTAMP DEFAULT NOW()
);
-- Index for listing sessions
CREATE INDEX IF NOT EXISTS idx_ocr_pipeline_sessions_created
ON ocr_pipeline_sessions (created_at DESC);
CREATE INDEX IF NOT EXISTS idx_ocr_pipeline_sessions_status
ON ocr_pipeline_sessions (status);

View File

@@ -0,0 +1,4 @@
-- Migration 003: Add row_result column for row geometry detection
-- Stores detected row geometries including header/footer classification
ALTER TABLE ocr_pipeline_sessions ADD COLUMN IF NOT EXISTS row_result JSONB;

View File

@@ -0,0 +1,4 @@
-- Migration 004: Add word_result column for OCR Pipeline Step 5
-- Stores the word recognition grid result (entries with english/german/example + bboxes)
ALTER TABLE ocr_pipeline_sessions ADD COLUMN IF NOT EXISTS word_result JSONB;

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,228 @@
"""
OCR Pipeline Session Store - PostgreSQL persistence for OCR pipeline sessions.
Replaces in-memory storage with database persistence.
See migrations/002_ocr_pipeline_sessions.sql for schema.
"""
import os
import uuid
import logging
import json
from typing import Optional, List, Dict, Any
import asyncpg
logger = logging.getLogger(__name__)
# Database configuration (same as vocab_session_store)
DATABASE_URL = os.getenv(
"DATABASE_URL",
"postgresql://breakpilot:breakpilot@postgres:5432/breakpilot_db"
)
# Connection pool (initialized lazily)
_pool: Optional[asyncpg.Pool] = None
async def get_pool() -> asyncpg.Pool:
"""Get or create the database connection pool."""
global _pool
if _pool is None:
_pool = await asyncpg.create_pool(DATABASE_URL, min_size=2, max_size=10)
return _pool
async def init_ocr_pipeline_tables():
"""Initialize OCR pipeline tables if they don't exist."""
pool = await get_pool()
async with pool.acquire() as conn:
tables_exist = await conn.fetchval("""
SELECT EXISTS (
SELECT FROM information_schema.tables
WHERE table_name = 'ocr_pipeline_sessions'
)
""")
if not tables_exist:
logger.info("Creating OCR pipeline tables...")
migration_path = os.path.join(
os.path.dirname(__file__),
"migrations/002_ocr_pipeline_sessions.sql"
)
if os.path.exists(migration_path):
with open(migration_path, "r") as f:
sql = f.read()
await conn.execute(sql)
logger.info("OCR pipeline tables created successfully")
else:
logger.warning(f"Migration file not found: {migration_path}")
else:
logger.debug("OCR pipeline tables already exist")
# =============================================================================
# SESSION CRUD
# =============================================================================
async def create_session_db(
session_id: str,
name: str,
filename: str,
original_png: bytes,
) -> Dict[str, Any]:
"""Create a new OCR pipeline session."""
pool = await get_pool()
async with pool.acquire() as conn:
row = await conn.fetchrow("""
INSERT INTO ocr_pipeline_sessions (
id, name, filename, original_png, status, current_step
) VALUES ($1, $2, $3, $4, 'active', 1)
RETURNING id, name, filename, status, current_step,
deskew_result, dewarp_result, column_result, row_result,
word_result, ground_truth, auto_shear_degrees,
created_at, updated_at
""", uuid.UUID(session_id), name, filename, original_png)
return _row_to_dict(row)
async def get_session_db(session_id: str) -> Optional[Dict[str, Any]]:
"""Get session metadata (without images)."""
pool = await get_pool()
async with pool.acquire() as conn:
row = await conn.fetchrow("""
SELECT id, name, filename, status, current_step,
deskew_result, dewarp_result, column_result, row_result,
word_result, ground_truth, auto_shear_degrees,
created_at, updated_at
FROM ocr_pipeline_sessions WHERE id = $1
""", uuid.UUID(session_id))
if row:
return _row_to_dict(row)
return None
async def get_session_image(session_id: str, image_type: str) -> Optional[bytes]:
"""Load a single image (BYTEA) from the session."""
column_map = {
"original": "original_png",
"deskewed": "deskewed_png",
"binarized": "binarized_png",
"dewarped": "dewarped_png",
}
column = column_map.get(image_type)
if not column:
return None
pool = await get_pool()
async with pool.acquire() as conn:
return await conn.fetchval(
f"SELECT {column} FROM ocr_pipeline_sessions WHERE id = $1",
uuid.UUID(session_id)
)
async def update_session_db(session_id: str, **kwargs) -> Optional[Dict[str, Any]]:
"""Update session fields dynamically."""
pool = await get_pool()
fields = []
values = []
param_idx = 1
allowed_fields = {
'name', 'filename', 'status', 'current_step',
'original_png', 'deskewed_png', 'binarized_png', 'dewarped_png',
'deskew_result', 'dewarp_result', 'column_result', 'row_result',
'word_result', 'ground_truth', 'auto_shear_degrees',
}
jsonb_fields = {'deskew_result', 'dewarp_result', 'column_result', 'row_result', 'word_result', 'ground_truth'}
for key, value in kwargs.items():
if key in allowed_fields:
fields.append(f"{key} = ${param_idx}")
if key in jsonb_fields and value is not None and not isinstance(value, str):
value = json.dumps(value)
values.append(value)
param_idx += 1
if not fields:
return await get_session_db(session_id)
# Always update updated_at
fields.append(f"updated_at = NOW()")
values.append(uuid.UUID(session_id))
async with pool.acquire() as conn:
row = await conn.fetchrow(f"""
UPDATE ocr_pipeline_sessions
SET {', '.join(fields)}
WHERE id = ${param_idx}
RETURNING id, name, filename, status, current_step,
deskew_result, dewarp_result, column_result, row_result,
word_result, ground_truth, auto_shear_degrees,
created_at, updated_at
""", *values)
if row:
return _row_to_dict(row)
return None
async def list_sessions_db(limit: int = 50) -> List[Dict[str, Any]]:
"""List all sessions (metadata only, no images)."""
pool = await get_pool()
async with pool.acquire() as conn:
rows = await conn.fetch("""
SELECT id, name, filename, status, current_step,
created_at, updated_at
FROM ocr_pipeline_sessions
ORDER BY created_at DESC
LIMIT $1
""", limit)
return [_row_to_dict(row) for row in rows]
async def delete_session_db(session_id: str) -> bool:
"""Delete a session."""
pool = await get_pool()
async with pool.acquire() as conn:
result = await conn.execute("""
DELETE FROM ocr_pipeline_sessions WHERE id = $1
""", uuid.UUID(session_id))
return result == "DELETE 1"
# =============================================================================
# HELPER
# =============================================================================
def _row_to_dict(row: asyncpg.Record) -> Dict[str, Any]:
"""Convert asyncpg Record to JSON-serializable dict."""
if row is None:
return {}
result = dict(row)
# UUID → string
for key in ['id', 'session_id']:
if key in result and result[key] is not None:
result[key] = str(result[key])
# datetime → ISO string
for key in ['created_at', 'updated_at']:
if key in result and result[key] is not None:
result[key] = result[key].isoformat()
# JSONB → parsed (asyncpg returns str for JSONB)
for key in ['deskew_result', 'dewarp_result', 'column_result', 'row_result', 'word_result', 'ground_truth']:
if key in result and result[key] is not None:
if isinstance(result[key], str):
result[key] = json.loads(result[key])
return result

View File

@@ -28,6 +28,13 @@ opencv-python-headless>=4.8.0
pytesseract>=0.3.10
Pillow>=10.0.0
# RapidOCR (PaddleOCR models on ONNX Runtime — works on ARM64 natively)
rapidocr
onnxruntime
# IPA pronunciation dictionary lookup (MIT license, bundled CMU dict ~134k words)
eng-to-ipa
# PostgreSQL (for metrics storage)
psycopg2-binary>=2.9.0
asyncpg>=0.29.0

View File

@@ -9,6 +9,9 @@ Tests cover:
- Stage 5: Layout analysis (content bounds, projection profiles, column detection)
- Stage 6: Multi-pass OCR region handling
- Stage 7: Line grouping and vocabulary matching
- Noise filter functions (_is_noise_tail_token, _clean_cell_text)
- Phonetic detection (_is_phonetic_only_text)
- Phonetic & continuation row merging
- Orchestrator (run_cv_pipeline)
DSGVO Note: All tests run locally with synthetic data. No external API calls.
@@ -21,6 +24,7 @@ from dataclasses import asdict
# Import module under test
from cv_vocab_pipeline import (
ColumnGeometry,
PageRegion,
VocabRow,
PipelineResult,
@@ -29,6 +33,12 @@ from cv_vocab_pipeline import (
create_ocr_image,
create_layout_image,
_find_content_bounds,
_filter_narrow_runs,
_build_margin_regions,
_detect_header_footer_gaps,
_detect_sub_columns,
_region_has_content,
_add_header_footer,
analyze_layout,
_group_words_into_lines,
match_lines_to_vocab,
@@ -36,6 +46,11 @@ from cv_vocab_pipeline import (
CV2_AVAILABLE,
TESSERACT_AVAILABLE,
CV_PIPELINE_AVAILABLE,
_is_noise_tail_token,
_clean_cell_text,
_is_phonetic_only_text,
_merge_phonetic_continuation_rows,
_merge_continuation_rows,
)
@@ -202,16 +217,28 @@ class TestDeskew:
@pytest.mark.skipif(not CV2_AVAILABLE, reason="OpenCV not available")
class TestDewarp:
"""Test dewarp (pass-through) stage."""
"""Test dewarp stage (returns (image, info) tuple)."""
def test_dewarp_passthrough(self, white_image):
"""Current dewarp should return the same image (pass-through)."""
def test_dewarp_returns_tuple(self, white_image):
"""dewarp_image must return (image, dewarp_info) tuple."""
result = dewarp_image(white_image)
np.testing.assert_array_equal(result, white_image)
assert isinstance(result, tuple)
assert len(result) == 2
img_out, info = result
assert isinstance(img_out, np.ndarray)
assert isinstance(info, dict)
assert "shear_degrees" in info
def test_dewarp_preserves_shape(self, text_like_image):
result = dewarp_image(text_like_image)
assert result.shape == text_like_image.shape
"""Output image should have same shape as input."""
img_out, _ = dewarp_image(text_like_image)
assert img_out.shape == text_like_image.shape
def test_dewarp_white_image_no_correction(self, white_image):
"""A uniform white image should get no shear correction."""
img_out, info = dewarp_image(white_image)
assert abs(info["shear_degrees"]) < 0.5
assert img_out.shape == white_image.shape
# =============================================
@@ -317,7 +344,8 @@ class TestLayoutAnalysis:
ocr_img = create_ocr_image(text_like_image)
layout_img = create_layout_image(text_like_image)
regions = analyze_layout(layout_img, ocr_img)
valid_types = {'column_en', 'column_de', 'column_example', 'header', 'footer'}
valid_types = {'column_en', 'column_de', 'column_example',
'header', 'footer', 'margin_top', 'margin_bottom'}
for r in regions:
assert r.type in valid_types, f"Unexpected region type: {r.type}"
@@ -561,6 +589,816 @@ class TestStageIntegration:
assert layout_img.shape[:2] == corrected.shape[:2]
# =============================================
# NOISE FILTER TESTS
# =============================================
class TestNoiseFilter:
"""Test _is_noise_tail_token for trailing OCR noise detection."""
# --- Tokens that should be KEPT (return False) ---
@pytest.mark.parametrize("token", [
# Compound words with hyphens
"money-saver",
"under-",
"well-known",
# Words with parenthesized parts (dictionary entries)
"Schild(chen)",
"(Salat-)Gurke",
"(auf)",
"(on)",
"selbst)",
"(wir",
"Tanz(veranstaltung)",
"(zer)brechen",
# Phonetic brackets
"serva]",
"['mani",
"[eg]",
"[maus]",
# Words with trailing punctuation
"cupcakes.",
"sister.",
"mice",
# Abbreviations
"e.g.",
"sth.",
"usw.",
"adj.",
# Ellipsis
"...",
"\u2026",
# Regular words
"the",
"cat",
"big",
"run",
"set",
"ago",
])
def test_keep_real_tokens(self, token):
"""Real words, dictionary punctuation, and phonetic brackets are kept."""
assert _is_noise_tail_token(token) is False, f"Should keep {token!r}"
# --- Tokens that should be FILTERED (return True) ---
@pytest.mark.parametrize("token", [
# Pure non-alpha
"B|",
"3d",
"x7",
")",
"|",
"@",
"3",
# Very short non-dictionary fragments
"ee",
"k",
"zz",
"qq",
# Empty
"",
" ",
])
def test_filter_noise_tokens(self, token):
"""OCR noise fragments are filtered."""
assert _is_noise_tail_token(token) is True, f"Should filter {token!r}"
class TestCleanCellText:
"""Test _clean_cell_text integration (full text → cleaned text)."""
def test_empty_returns_empty(self):
assert _clean_cell_text("") == ""
assert _clean_cell_text(" ") == ""
def test_real_word_unchanged(self):
assert _clean_cell_text("cupcakes") == "cupcakes"
def test_strips_trailing_noise(self):
"""Trailing noise tokens should be removed."""
result = _clean_cell_text("cupcakes B|")
assert result == "cupcakes"
def test_keeps_trailing_real_word(self):
"""Trailing real words should be kept."""
result = _clean_cell_text("big cat")
assert result == "big cat"
def test_abbreviation_kept(self):
"""Known abbreviations should not be cleared."""
result = _clean_cell_text("e.g.")
assert result == "e.g."
def test_pure_garbage_cleared(self):
"""OCR garbage without real words should be cleared."""
result = _clean_cell_text("3d |x")
assert result == ""
def test_compound_word_preserved(self):
"""Compound words with hyphens should be preserved."""
result = _clean_cell_text("money-saver")
assert result == "money-saver"
def test_parenthesized_word_preserved(self):
result = _clean_cell_text("(Salat-)Gurke")
assert result == "(Salat-)Gurke"
def test_multiple_trailing_noise(self):
"""Multiple trailing noise tokens should all be removed."""
result = _clean_cell_text("achieve 3 |")
assert result == "achieve"
class TestPhoneticOnlyText:
"""Test _is_phonetic_only_text for phonetic transcription detection."""
@pytest.mark.parametrize("text,expected", [
# Phonetic-only patterns → True
("['mani serva]", True),
("[dɑːns]", True),
("[\"a:mand]", True),
("['ːkʃɒp]", True),
# serva] has 5 alpha chars after bracket removal → NOT phonetic-only
("serva]", False),
# NOT phonetic-only → False
("almond ['a:mand]", False),
("Mandel", False),
("cupcakes", False),
("", False),
("achieve", False),
("money-saver ['mani]", False),
])
def test_phonetic_detection(self, text, expected):
assert _is_phonetic_only_text(text) is expected, \
f"_is_phonetic_only_text({text!r}) should be {expected}"
class TestMergePhoneticContinuationRows:
"""Test _merge_phonetic_continuation_rows for phonetic row merging."""
def test_empty_list(self):
assert _merge_phonetic_continuation_rows([]) == []
def test_single_entry(self):
entries = [{"english": "cat", "german": "Katze", "example": ""}]
result = _merge_phonetic_continuation_rows(entries)
assert len(result) == 1
assert result[0]["english"] == "cat"
def test_merges_phonetic_row(self):
"""Phonetic-only row should merge into previous entry."""
entries = [
{"english": "money-saver", "german": "Sparfuchs", "example": "", "row_index": 0},
{"english": "['mani serva]", "german": "", "example": "", "row_index": 1},
]
result = _merge_phonetic_continuation_rows(entries)
assert len(result) == 1
assert result[0]["english"] == "money-saver ['mani serva]"
assert result[0]["german"] == "Sparfuchs"
def test_no_merge_when_de_present(self):
"""Row with DE text should NOT be merged even if EN looks phonetic."""
entries = [
{"english": "cat", "german": "Katze", "example": ""},
{"english": "[kæt]", "german": "some text", "example": ""},
]
result = _merge_phonetic_continuation_rows(entries)
assert len(result) == 2
def test_no_merge_regular_rows(self):
"""Normal vocab rows should not be merged."""
entries = [
{"english": "cat", "german": "Katze", "example": ""},
{"english": "dog", "german": "Hund", "example": ""},
]
result = _merge_phonetic_continuation_rows(entries)
assert len(result) == 2
def test_merges_example_too(self):
"""If phonetic row has example text, it should merge into previous."""
entries = [
{"english": "dance", "german": "tanzen", "example": "", "row_index": 0},
{"english": "[dɑːns]", "german": "", "example": "Let's dance.", "row_index": 1},
]
result = _merge_phonetic_continuation_rows(entries)
assert len(result) == 1
assert result[0]["english"] == "dance [dɑːns]"
assert result[0]["example"] == "Let's dance."
class TestMergeContinuationRows:
"""Test _merge_continuation_rows for multi-line entry merging."""
def test_empty_list(self):
assert _merge_continuation_rows([]) == []
def test_no_merge_independent_rows(self):
"""Rows with both EN and DE should not be merged."""
entries = [
{"english": "cat", "german": "Katze", "example": "", "row_index": 0},
{"english": "dog", "german": "Hund", "example": "", "row_index": 1},
]
result = _merge_continuation_rows(entries)
assert len(result) == 2
def test_merge_lowercase_continuation(self):
"""Lowercase EN with empty DE should merge into previous."""
entries = [
{"english": "to put up", "german": "aufstellen", "example": "", "row_index": 0},
{"english": "with sth.", "german": "", "example": "", "row_index": 1},
]
result = _merge_continuation_rows(entries)
assert len(result) == 1
assert result[0]["english"] == "to put up with sth."
assert result[0]["german"] == "aufstellen"
def test_no_merge_uppercase_start(self):
"""EN starting with uppercase and empty DE is likely its own entry, not a continuation."""
entries = [
{"english": "cat", "german": "Katze", "example": "", "row_index": 0},
{"english": "Dog", "german": "", "example": "", "row_index": 1},
]
result = _merge_continuation_rows(entries)
assert len(result) == 2
def test_no_merge_when_previous_ends_with_period(self):
"""If previous entry ends with sentence terminator, next is not continuation."""
entries = [
{"english": "That's great.", "german": "Das ist toll.", "example": "", "row_index": 0},
{"english": "really nice", "german": "", "example": "", "row_index": 1},
]
result = _merge_continuation_rows(entries)
assert len(result) == 2
def test_no_merge_long_text(self):
"""Text with 4+ words is likely an example sentence, not continuation."""
entries = [
{"english": "achieve", "german": "erreichen", "example": "", "row_index": 0},
{"english": "she achieved her goals", "german": "", "example": "", "row_index": 1},
]
result = _merge_continuation_rows(entries)
assert len(result) == 2
def test_first_entry_not_merged(self):
"""First entry with empty DE should not crash (no previous)."""
entries = [
{"english": "something", "german": "", "example": "", "row_index": 0},
{"english": "cat", "german": "Katze", "example": "", "row_index": 1},
]
result = _merge_continuation_rows(entries)
assert len(result) == 2
# =============================================
# Test: Content-Bounds Scan-Artifact Filtering
# =============================================
class TestContentBoundsFiltering:
"""Test that _find_content_bounds filters narrow scan artifacts."""
def test_thin_vertical_line_ignored(self):
"""A 2px black line at the left edge should not pull left_x leftward."""
inv = np.zeros((400, 600), dtype=np.uint8)
# Main content block in the middle
inv[50:350, 100:550] = 255
# 2px thin vertical scan artifact at x=5..6
inv[50:350, 5:7] = 255
left, right, top, bottom = _find_content_bounds(inv)
# left_x must be near 100 (the real content), not near 5
assert left >= 90, f"left_x={left} should be >=90 (near real content, not artifact)"
def test_thick_content_preserved(self):
"""A 50px wide text block is real content and must not be filtered."""
inv = np.zeros((400, 600), dtype=np.uint8)
inv[50:350, 80:130] = 255 # 50px wide block
inv[50:350, 200:500] = 255 # wider block
left, right, top, bottom = _find_content_bounds(inv)
assert left <= 82, f"left_x={left} should be <=82 (50px block is real content)"
def test_no_artifacts_unchanged(self):
"""Normal image without artifacts: bounds should match content."""
inv = np.zeros((400, 600), dtype=np.uint8)
inv[100:300, 50:550] = 255
left, right, top, bottom = _find_content_bounds(inv)
assert left <= 52
assert right >= 548
assert top <= 105
assert bottom >= 295
def test_right_edge_artifact_ignored(self):
"""A thin vertical line at the right edge should not pull right_x rightward."""
inv = np.zeros((400, 600), dtype=np.uint8)
inv[50:350, 50:500] = 255 # real content
inv[50:350, 595:598] = 255 # 3px artifact at right edge
left, right, top, bottom = _find_content_bounds(inv)
assert right <= 510, f"right_x={right} should be <=510, ignoring right-edge artifact"
def test_horizontal_line_ignored(self):
"""A thin horizontal line at the top should not pull top_y upward."""
inv = np.zeros((400, 600), dtype=np.uint8)
inv[100:350, 50:550] = 255 # real content
inv[2:4, 50:550] = 255 # 2px horizontal artifact at top
left, right, top, bottom = _find_content_bounds(inv)
assert top >= 90, f"top_y={top} should be >=90 (ignoring thin top line)"
class TestFilterNarrowRuns:
"""Test the _filter_narrow_runs helper directly."""
def test_removes_short_run(self):
mask = np.array([False, True, True, False, True, True, True, True, True, False])
result = _filter_narrow_runs(mask, min_width=3)
# The 2-wide run at indices 1-2 should be removed
assert not result[1]
assert not result[2]
# The 5-wide run at indices 4-8 should remain
assert result[4]
assert result[8]
def test_keeps_wide_run(self):
mask = np.array([True] * 10)
result = _filter_narrow_runs(mask, min_width=5)
assert all(result)
def test_all_narrow(self):
mask = np.array([True, True, False, True, False])
result = _filter_narrow_runs(mask, min_width=3)
assert not any(result)
# =============================================
# Test: Margin Regions
# =============================================
class TestMarginRegions:
"""Test _build_margin_regions and margin integration."""
def test_margin_left_created(self):
"""When left_x > 5, a margin_left region should be created."""
existing = [
PageRegion(type='column_en', x=100, y=50, width=200, height=300),
PageRegion(type='column_de', x=320, y=50, width=200, height=300),
]
margins = _build_margin_regions(existing, left_x=100, right_x=520,
img_w=600, top_y=50, content_h=300)
left_margins = [m for m in margins if m.type == 'margin_left']
assert len(left_margins) == 1
ml = left_margins[0]
assert ml.x == 0
assert ml.width == 100
def test_margin_right_created(self):
"""When there's space after the last column, margin_right should be created."""
existing = [
PageRegion(type='column_en', x=50, y=50, width=200, height=300),
PageRegion(type='column_de', x=260, y=50, width=200, height=300),
]
# last_col_end = 260 + 200 = 460, img_w = 600 → gap = 140
margins = _build_margin_regions(existing, left_x=50, right_x=460,
img_w=600, top_y=50, content_h=300)
right_margins = [m for m in margins if m.type == 'margin_right']
assert len(right_margins) == 1
mr = right_margins[0]
assert mr.x == 460
assert mr.width == 140
def test_no_margin_when_flush(self):
"""When columns are flush with the image edges, no margins should appear."""
existing = [
PageRegion(type='column_en', x=0, y=0, width=300, height=400),
PageRegion(type='column_de', x=300, y=0, width=300, height=400),
]
margins = _build_margin_regions(existing, left_x=0, right_x=600,
img_w=600, top_y=0, content_h=400)
assert len(margins) == 0
def test_margins_in_skip_types(self):
"""Verify margin types are in the skip set used by build_cell_grid."""
skip = {'column_ignore', 'header', 'footer', 'margin_top', 'margin_bottom', 'page_ref', 'margin_left', 'margin_right'}
assert 'margin_left' in skip
assert 'margin_right' in skip
def test_margin_confidence_and_method(self):
"""Margin regions should have confidence 1.0 and method 'content_bounds'."""
existing = [PageRegion(type='column_en', x=80, y=20, width=400, height=500)]
margins = _build_margin_regions(existing, left_x=80, right_x=480,
img_w=600, top_y=20, content_h=500)
for m in margins:
assert m.classification_confidence == 1.0
assert m.classification_method == 'content_bounds'
# =============================================
# Header/Footer Gap Detection
# =============================================
class TestHeaderFooterGapDetection:
"""Tests for _detect_header_footer_gaps()."""
def _make_inv(self, height: int, width: int, bands: list) -> np.ndarray:
"""Create an inverted binary image with white horizontal bands.
Args:
height: Image height.
width: Image width.
bands: List of (y_start, y_end) tuples where pixels are white (255).
"""
inv = np.zeros((height, width), dtype=np.uint8)
for y1, y2 in bands:
inv[y1:y2, :] = 255
return inv
def _make_body_with_lines(self, h, w, body_start, body_end,
line_h=15, gap_h=12):
"""Create bands simulating text lines with inter-line gaps.
gap_h must be large enough to survive smoothing (kernel ~ h//200).
"""
bands = []
y = body_start
while y + line_h <= body_end:
bands.append((y, y + line_h))
y += line_h + gap_h
return bands
def test_header_gap_detected(self):
"""Content at top + large gap + main body → header_y at the gap."""
h, w = 2000, 800
# Header content at rows 20-80
bands = [(20, 80)]
# Large gap 80-300 (220px) — much larger than 12px line gaps
# Body lines from 300 to ~1990 (extends near bottom, no footer gap)
bands += self._make_body_with_lines(h, w, 300, 1990)
inv = self._make_inv(h, w, bands)
header_y, footer_y = _detect_header_footer_gaps(inv, w, h)
assert header_y is not None
assert 80 <= header_y <= 310
def test_footer_gap_detected(self):
"""Main body + large gap + page number → footer_y at the gap."""
h, w = 2000, 800
# Body lines from 10 to 1600 (starts near top, no header gap)
bands = self._make_body_with_lines(h, w, 10, 1600)
# Large gap 1600-1880 (280px)
# Page number 1880-1920
bands.append((1880, 1920))
inv = self._make_inv(h, w, bands)
header_y, footer_y = _detect_header_footer_gaps(inv, w, h)
assert footer_y is not None
assert 1580 <= footer_y <= 1890
def test_both_header_and_footer(self):
"""Header + gap + body lines + gap + footer → both detected."""
h, w = 2000, 800
# Header 10-60
bands = [(10, 60)]
# Large gap 60-250 (190px)
# Body lines from 250 to 1700
bands += self._make_body_with_lines(h, w, 250, 1700)
# Large gap 1700-1900 (200px)
# Footer 1900-1970
bands.append((1900, 1970))
inv = self._make_inv(h, w, bands)
header_y, footer_y = _detect_header_footer_gaps(inv, w, h)
assert header_y is not None
assert footer_y is not None
assert 60 <= header_y <= 260
assert 1690 <= footer_y <= 1910
def test_no_gaps_returns_none(self):
"""Uniform content across the page → (None, None)."""
h, w = 1000, 800
# Content across entire height
inv = self._make_inv(h, w, [(0, 1000)])
header_y, footer_y = _detect_header_footer_gaps(inv, w, h)
assert header_y is None
assert footer_y is None
def test_small_gaps_ignored(self):
"""Gaps smaller than 2x median should be ignored."""
h, w = 1000, 800
# Many small, evenly-spaced gaps (like line spacing) — no large outlier
bands = []
for row_start in range(0, 1000, 20):
bands.append((row_start, row_start + 15)) # 15px content, 5px gap
inv = self._make_inv(h, w, bands)
header_y, footer_y = _detect_header_footer_gaps(inv, w, h)
# All gaps are equal size, none > 2x median → no header/footer
assert header_y is None
assert footer_y is None
def test_edge_gaps_ignored_dewarp_padding(self):
"""Trailing gap at bottom edge (dewarp padding) should not be detected as footer."""
h, w = 2000, 800
# Body lines from 10 to 1700
bands = self._make_body_with_lines(h, w, 10, 1700)
# Gap from 1700 to 2000 = bottom edge padding (no content after)
inv = self._make_inv(h, w, bands)
header_y, footer_y = _detect_header_footer_gaps(inv, w, h)
# The trailing gap touches the image edge → not a valid separator
assert footer_y is None
class TestRegionContentCheck:
"""Tests for _region_has_content() and _add_header_footer() type selection."""
def _make_inv(self, height: int, width: int, bands: list) -> np.ndarray:
inv = np.zeros((height, width), dtype=np.uint8)
for y1, y2 in bands:
inv[y1:y2, :] = 255
return inv
def test_region_with_text_has_content(self):
"""Strip with ink → True."""
inv = self._make_inv(1000, 800, [(10, 50)])
assert _region_has_content(inv, 0, 100) is True
def test_empty_region_no_content(self):
"""Strip without ink → False."""
inv = self._make_inv(1000, 800, [(500, 600)])
assert _region_has_content(inv, 0, 100) is False
def test_header_with_text_is_header(self):
"""Top region with text → type='header' (via content bounds fallback)."""
h, w = 1000, 800
# Header text at 20-60, body starts at 200
inv = self._make_inv(h, w, [(20, 60), (200, 900)])
regions: list = []
# Simulate content bounds detecting body start at y=200
_add_header_footer(regions, top_y=200, bottom_y=h, img_w=w, img_h=h, inv=inv)
top_regions = [r for r in regions if r.type in ('header', 'margin_top')]
assert len(top_regions) == 1
assert top_regions[0].type == 'header' # text at 20-60 → header
def test_empty_top_is_margin_top(self):
"""Top region without text → type='margin_top'."""
h, w = 1000, 800
# Content only in body area (200-900), nothing in top 200px
inv = self._make_inv(h, w, [(200, 900)])
regions: list = []
# Simulate top_y=200 from content bounds
_add_header_footer(regions, top_y=200, bottom_y=h, img_w=w, img_h=h, inv=inv)
top_regions = [r for r in regions if r.type in ('header', 'margin_top')]
assert len(top_regions) == 1
assert top_regions[0].type == 'margin_top'
def test_empty_bottom_is_margin_bottom(self):
"""Bottom region without text → type='margin_bottom'."""
h, w = 1000, 800
# Content only in top/body (50-700), nothing below 700
inv = self._make_inv(h, w, [(50, 700)])
regions: list = []
_add_header_footer(regions, top_y=50, bottom_y=700, img_w=w, img_h=h, inv=inv)
bottom_regions = [r for r in regions if r.type in ('footer', 'margin_bottom')]
assert len(bottom_regions) == 1
assert bottom_regions[0].type == 'margin_bottom'
def test_footer_with_page_number_is_footer(self):
"""Bottom region with page number text → type='footer'."""
h, w = 1000, 800
# Body 50-700, page number at 900-930
inv = self._make_inv(h, w, [(50, 700), (900, 930)])
regions: list = []
_add_header_footer(regions, top_y=50, bottom_y=700, img_w=w, img_h=h, inv=inv)
bottom_regions = [r for r in regions if r.type in ('footer', 'margin_bottom')]
assert len(bottom_regions) == 1
assert bottom_regions[0].type == 'footer'
# =============================================
# Sub-Column Detection Tests
# =============================================
class TestSubColumnDetection:
"""Tests for _detect_sub_columns() left-edge alignment detection."""
def _make_word(self, left: int, text: str = "word", conf: int = 90) -> dict:
return {'left': left, 'top': 100, 'width': 50, 'height': 20,
'text': text, 'conf': conf}
def _make_geo(self, x: int, width: int, words: list, content_w: int = 1000) -> ColumnGeometry:
return ColumnGeometry(
index=0, x=x, y=50, width=width, height=500,
word_count=len(words), words=words,
width_ratio=width / content_w,
)
def test_sub_column_split_page_refs(self):
"""3 page-refs left + 40 vocab words right → split into 2.
The leftmost bin with >= 10% of words (>= 5) is the vocab bin
at left=250, so the 3 page-refs are outliers.
"""
content_w = 1000
page_words = [self._make_word(100, f"p.{59+i}") for i in range(3)]
vocab_words = [self._make_word(250, f"word{i}") for i in range(40)]
all_words = page_words + vocab_words
geo = self._make_geo(x=80, width=300, words=all_words, content_w=content_w)
result = _detect_sub_columns([geo], content_w)
assert len(result) == 2, f"Expected 2 columns, got {len(result)}"
left_col = result[0]
right_col = result[1]
assert left_col.x < right_col.x
assert left_col.word_count == 3
assert right_col.word_count == 40
assert left_col.index == 0
assert right_col.index == 1
def test_sub_column_split_exclamation_marks(self):
"""5 '!' (misread as I/|) left + 80 example words → split into 2.
Mirrors the real-world case where red ! marks are OCR'd as I, |, B, 1
at a position slightly left of the example sentence start.
"""
content_w = 1500
bang_words = [self._make_word(950 + i, chr(ord('I')), conf=60) for i in range(5)]
example_words = [self._make_word(975 + (i * 3), f"word{i}") for i in range(80)]
all_words = bang_words + example_words
geo = self._make_geo(x=940, width=530, words=all_words, content_w=content_w)
result = _detect_sub_columns([geo], content_w)
assert len(result) == 2
assert result[0].word_count == 5
assert result[1].word_count == 80
def test_no_split_uniform_alignment(self):
"""All words aligned at same position → no change."""
content_w = 1000
words = [self._make_word(200, f"word{i}") for i in range(15)]
geo = self._make_geo(x=180, width=300, words=words, content_w=content_w)
result = _detect_sub_columns([geo], content_w)
assert len(result) == 1
assert result[0].word_count == 15
def test_no_split_narrow_column(self):
"""Narrow column (width_ratio < 0.15) → no split attempted."""
content_w = 1000
words = [self._make_word(50, "a")] * 3 + [self._make_word(120, "b")] * 10
geo = self._make_geo(x=40, width=140, words=words, content_w=content_w)
result = _detect_sub_columns([geo], content_w)
assert len(result) == 1
def test_no_split_balanced_clusters(self):
"""Both clusters similarly sized (ratio >= 0.35) → no split."""
content_w = 1000
left_words = [self._make_word(100, f"a{i}") for i in range(8)]
right_words = [self._make_word(300, f"b{i}") for i in range(12)]
all_words = left_words + right_words
geo = self._make_geo(x=80, width=400, words=all_words, content_w=content_w)
result = _detect_sub_columns([geo], content_w)
assert len(result) == 1
def test_sub_column_reindexing(self):
"""After split, indices are correctly 0, 1, 2 across all columns."""
content_w = 1000
# First column: no split (all words at same alignment)
words1 = [self._make_word(50, f"de{i}") for i in range(10)]
geo1 = ColumnGeometry(index=0, x=30, y=50, width=200, height=500,
word_count=10, words=words1, width_ratio=0.2)
# Second column: will split (3 outliers + 40 main)
page_words = [self._make_word(400, f"p.{i}") for i in range(3)]
en_words = [self._make_word(550, f"en{i}") for i in range(40)]
geo2 = ColumnGeometry(index=1, x=380, y=50, width=300, height=500,
word_count=43, words=page_words + en_words, width_ratio=0.3)
result = _detect_sub_columns([geo1, geo2], content_w)
assert len(result) == 3
assert [g.index for g in result] == [0, 1, 2]
assert result[0].word_count == 10
assert result[1].word_count == 3
assert result[2].word_count == 40
def test_no_split_too_few_words(self):
"""Column with fewer than 5 words → no split attempted."""
content_w = 1000
words = [self._make_word(100, "a"), self._make_word(300, "b"),
self._make_word(300, "c"), self._make_word(300, "d")]
geo = self._make_geo(x=80, width=300, words=words, content_w=content_w)
result = _detect_sub_columns([geo], content_w)
assert len(result) == 1
def test_no_split_single_minority_word(self):
"""Only 1 word left of column start → no split (need >= 2)."""
content_w = 1000
minority = [self._make_word(100, "p.59")]
majority = [self._make_word(300, f"w{i}") for i in range(30)]
geo = self._make_geo(x=80, width=350, words=minority + majority, content_w=content_w)
result = _detect_sub_columns([geo], content_w)
assert len(result) == 1
def test_sub_column_split_with_left_x_offset(self):
"""Word 'left' values are relative to left_x; geo.x is absolute.
Real-world scenario: left_x=195, EN column at geo.x=310.
Page refs at relative left=115-157, vocab words at relative left=216.
Without left_x, split_x would be ~202 (< geo.x=310) → negative width → no split.
With left_x=195, split_abs = 202 + 195 = 397, which is between geo.x(310)
and geo.x+geo.width(748) → valid split.
"""
content_w = 1469
left_x = 195
page_refs = [self._make_word(115, "p.59"), self._make_word(157, "p.60"),
self._make_word(157, "p.61")]
vocab = [self._make_word(216, f"word{i}") for i in range(40)]
all_words = page_refs + vocab
geo = self._make_geo(x=310, width=438, words=all_words, content_w=content_w)
result = _detect_sub_columns([geo], content_w, left_x=left_x)
assert len(result) == 2, f"Expected 2 columns, got {len(result)}"
assert result[0].word_count == 3
assert result[1].word_count == 40
class TestCellsToVocabEntriesPageRef:
"""Test that page_ref cells are mapped to source_page field."""
def test_page_ref_mapped_to_source_page(self):
"""Cell with col_type='page_ref' → source_page field populated."""
from cv_vocab_pipeline import _cells_to_vocab_entries
cells = [
{
'row_index': 0,
'col_type': 'column_en',
'text': 'hello',
'bbox_pct': {'x': 10, 'y': 10, 'w': 30, 'h': 5},
'confidence': 95.0,
'ocr_engine': 'tesseract',
},
{
'row_index': 0,
'col_type': 'column_de',
'text': 'hallo',
'bbox_pct': {'x': 40, 'y': 10, 'w': 30, 'h': 5},
'confidence': 90.0,
'ocr_engine': 'tesseract',
},
{
'row_index': 0,
'col_type': 'page_ref',
'text': 'p.59',
'bbox_pct': {'x': 5, 'y': 10, 'w': 5, 'h': 5},
'confidence': 80.0,
'ocr_engine': 'tesseract',
},
]
columns_meta = [
{'type': 'column_en'}, {'type': 'column_de'}, {'type': 'page_ref'},
]
entries = _cells_to_vocab_entries(cells, columns_meta)
assert len(entries) == 1
assert entries[0]['english'] == 'hello'
assert entries[0]['german'] == 'hallo'
assert entries[0]['source_page'] == 'p.59'
assert entries[0]['bbox_ref'] == {'x': 5, 'y': 10, 'w': 5, 'h': 5}
def test_no_page_ref_defaults_empty(self):
"""Without page_ref cell, source_page defaults to empty string."""
from cv_vocab_pipeline import _cells_to_vocab_entries
cells = [
{
'row_index': 0,
'col_type': 'column_en',
'text': 'world',
'bbox_pct': {'x': 10, 'y': 10, 'w': 30, 'h': 5},
'confidence': 95.0,
'ocr_engine': 'tesseract',
},
]
columns_meta = [{'type': 'column_en'}]
entries = _cells_to_vocab_entries(cells, columns_meta)
assert len(entries) == 1
assert entries[0]['source_page'] == ''
assert entries[0]['bbox_ref'] is None
# =============================================
# RUN TESTS
# =============================================

View File

@@ -65,10 +65,12 @@ nav:
- BYOEH Architektur: services/klausur-service/BYOEH-Architecture.md
- BYOEH Developer Guide: services/klausur-service/BYOEH-Developer-Guide.md
- NiBiS Pipeline: services/klausur-service/NiBiS-Ingestion-Pipeline.md
- OCR Pipeline: services/klausur-service/OCR-Pipeline.md
- OCR Labeling: services/klausur-service/OCR-Labeling-Spec.md
- OCR Vergleich: services/klausur-service/OCR-Compare.md
- RAG Admin: services/klausur-service/RAG-Admin-Spec.md
- Worksheet Editor: services/klausur-service/Worksheet-Editor-Architecture.md
- Chunk-Browser: services/klausur-service/Chunk-Browser.md
- Voice-Service:
- Uebersicht: services/voice-service/index.md
- Agent-Core: