ElebobaandClaude Opus 5.5
6a77831603
CI / ruff · pytest (pull_request) Successful in 1m30s
- docs/source/rag: PDF sources (KR PTSD 2023, KR depression 2024, meds module spec), pdf->txt converter (tables as markdown, page markers, hand-transcribed figures), word-level check against pdftotext; fixes HAM-D/MADRS fake headers and split rows - config/rag: corpus txt + sources.yaml (documents, audiences, retrieval params, synonyms) - app/llm/rag: Snowball stemmer, section-aware chunker with page citations, BM25 + synonym expansion + densest-passage reranking, extractive answer() with coverage gate, patient dose/prescribing filter, optional LLM rephrasing rejected when it adds content; CLI python -m app.llm.rag - tests: gold retrieval set (hit@3 >= 15/18), off-topic, safety, faithfulness - ADR-011 Refs #60 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
30 lines
1001 B
Python
30 lines
1001 B
Python
"""Сверка txt с PDF: какие слова из pdftotext не попали в txt (с учётом количества)."""
|
|
import re
|
|
import subprocess
|
|
import sys
|
|
from collections import Counter
|
|
from pathlib import Path
|
|
|
|
WORD = re.compile(r"[\wё]+", re.I)
|
|
|
|
|
|
def words(text: str) -> Counter[str]:
|
|
text = re.sub(r"(\w)-\s*\n\s*(\w)", r"\1\2", text) # переносы
|
|
return Counter(w.lower() for w in WORD.findall(text))
|
|
|
|
|
|
args = sys.argv[1:]
|
|
out = None
|
|
if "--out" in args:
|
|
i = args.index("--out")
|
|
out = Path(args[i + 1])
|
|
del args[i : i + 2]
|
|
for arg in args:
|
|
pdf = Path(arg)
|
|
ref = words(subprocess.run(["pdftotext", str(pdf), "-"], capture_output=True, text=True).stdout)
|
|
got = words(((out / pdf.name) if out else pdf).with_suffix(".txt").read_text())
|
|
lost = ref - got
|
|
total = sum(ref.values())
|
|
print(f"{pdf.name}: слов {total}, потеряно {sum(lost.values())} ({sum(lost.values()) / total:.2%})")
|
|
print(" ", lost.most_common(40))
|