"The pilot went well" is a sentence that tells an engineer, a lawyer, or a CFO nothing. Well, how? Were the answers accurate? All 50 control questions or just 30? Was the system tested on questions users actually ask, or on the ones that are convenient to show in a demo?
The difference between "we liked it" and "it passed acceptance" is a metric matrix, not a subjective impression from a few good-looking answers. You need to separately check: was the right document found, is the citation correct, does the text fully answer the question, does the model add anything of its own, and does the answer arrive within an acceptable time.
RAG evaluation surveys — Yu et al. (2024), Gao et al. (2023) — split these levels out explicitly: retrieval quality (was the right passage found), generation quality, i.e. faithfulness (how true the answer is to the retrieved context), and end-to-end usefulness for the user, which is not reducible to the first two and needs its own check.
Latency sits on this list on its own because it is not a technical detail but a condition of use: an accurate answer that arrives in 30 seconds may simply not fit a dispatcher's or field engineer's workflow — and then the system stops being opened at all, no matter how accurate it is.
Before launching a pilot, three things are worth putting on paper rather than keeping in someone's head: a list of 30-50 control questions with a known correct source; an acceptable latency threshold for the specific use scenario; and an explicit rule for what counts as a valid "insufficient evidence" answer — instead of sorting that out after the fact, once the system has already made something up.
The difference between an ordinary demo and a Knovium pilot is that here you can see in advance exactly where the system fell short: a log of failures and partial answers is kept alongside successful ones, and three answer states — found, partially found, insufficient evidence — are recorded for every control question, not just the ones that turned out well.