A common mistake in evaluating RAG: a team tests the system on the ten cleanest PDFs in the archive, gets great answers, and signs a contract for the whole archive based on that. A month into production, accuracy drops by half, and no one can explain why. The cause is usually not the model — it's that the "rest" of the archive is largely scans, tables with merged cells, and documents with no text layer at all.
RAG breaks not at answer generation but several steps earlier — at text extraction. A noisy scan, a table with a complex structure, or an engineering drawing can turn a document that is precise in content into a weak index before retrieval ever runs.
Research backs this up: OHRBench showed that OCR quality cascades through the whole pipeline — retrieval gets noise instead of text, ranking compares incomplete passages, and generation confidently writes an answer from damaged data. HD-RAG separately found that tables with merged cells need dedicated, table-aware parsing rather than line-by-line OCR. And the UDA benchmark on real-world documents shows that ordinary multi-column PDFs with footnotes score noticeably worse than a clean text corpus — even with the same model.
The direct consequence: ten clean PDFs say nothing about whether the whole archive is ready. A meaningful pilot has to include not the best documents but the typical ones — scans, tables, and multi-page attachments that users will actually ask about.
Before launching a pilot, it is worth honestly assessing the archive on four points: what share is scans with no text layer; are there tables with a complex structure; do page numbers survive conversion; can you cite a specific table row instead of only the file as a whole. "We don't know" on any of these is a signal that the archive assessment needs to happen before the pilot, not after it.
Knovium makes this assessment an explicit part of preparation: every document in the archive is tagged by source-quality class — "clean text", "PDF", "scan", "complex table" — and that tagging is visible to the client before the system starts answering questions, not surfaced as an excuse after the first weak answers.