fix(ocr-pipeline): overlap-based word assignment and empty row filtering
Some checks failed
CI / go-lint (push) Has been skipped
CI / python-lint (push) Has been skipped
CI / nodejs-lint (push) Has been skipped
CI / test-go-school (push) Successful in 1m14s
CI / test-go-edu-search (push) Successful in 26s
CI / test-python-klausur (push) Failing after 1m55s
CI / test-python-agent-core (push) Successful in 17s
CI / test-nodejs-website (push) Successful in 17s
Some checks failed
CI / go-lint (push) Has been skipped
CI / python-lint (push) Has been skipped
CI / nodejs-lint (push) Has been skipped
CI / test-go-school (push) Successful in 1m14s
CI / test-go-edu-search (push) Successful in 26s
CI / test-python-klausur (push) Failing after 1m55s
CI / test-python-agent-core (push) Successful in 17s
CI / test-nodejs-website (push) Successful in 17s
1. Word-to-column assignment now uses overlap-based matching instead of center-point matching. This fixes narrow page_ref columns losing their last digit (e.g. "p.59" → "p.5") when the digit's center falls slightly past the midpoint boundary into the next column. 2. Post-OCR empty row filter: rows where ALL cells have empty text are removed after OCR. This catches inter-row gaps that had stray Tesseract artifacts giving word_count > 0 but no actual content. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -1291,6 +1291,18 @@ async def _word_stream_generator(
|
||||
if columns_meta is None:
|
||||
columns_meta = []
|
||||
|
||||
# Post-OCR: remove rows where ALL cells are empty (inter-row gaps
|
||||
# that had stray Tesseract artifacts giving word_count > 0).
|
||||
rows_with_text: set = set()
|
||||
for c in all_cells:
|
||||
if c.get("text", "").strip():
|
||||
rows_with_text.add(c["row_index"])
|
||||
before_filter = len(all_cells)
|
||||
all_cells = [c for c in all_cells if c["row_index"] in rows_with_text]
|
||||
empty_rows_removed = (before_filter - len(all_cells)) // max(n_cols, 1)
|
||||
if empty_rows_removed > 0:
|
||||
logger.info(f"SSE: removed {empty_rows_removed} all-empty rows after OCR")
|
||||
|
||||
used_engine = all_cells[0].get("ocr_engine", "tesseract") if all_cells else engine
|
||||
|
||||
word_result = {
|
||||
|
||||
Reference in New Issue
Block a user