Measured, not claimed.
How often the engine is right, on examples labeled by people. The same scripts run on every engine change, so accuracy can only go up.
40 hand-labeled examples
5 numeric drift, 5 fabricated entity, 5 contradiction, 8 faithful, 5 meaning error, 6 instruction breach, 3 unsupported addition, 3 clean with instructions. A hallucination means the answer contains at least one claim the source contradicts or does not support.
| Configuration | Precision | Recall | Balanced accuracy | False alarms | Failure types found | Instruction checks |
|---|---|---|---|---|---|---|
| Rules only heuristic-1.3 | 100% | 70.8% | 85.4% | 0 | 100% | 100% |
| Rules + HHEM (production) heuristic-1.3 | 100% | 95.8% | 97.9% | 0 | 100% | 100% |
Known misses, Rules only (7)
- meaning error (meaning-1): missed hallucination
- meaning error (meaning-2): missed hallucination
- meaning error (meaning-3): missed hallucination
- meaning error (meaning-4): missed hallucination
- meaning error (meaning-5): missed hallucination
- instruction breach (instr-4): missed hallucination
- unsupported addition (added-2): missed hallucination
Known misses, Rules + HHEM (production) (1)
- meaning error (meaning-5): missed hallucination
Meaning errors such as role reversals or before/after swaps share every word with the source, so word-level rules cannot see them; the HHEM model and the optional LLM judge are there for those.
LLM-AggreFact
The standard public test of grounded fact-checking: real documents and claims from more than ten datasets (summaries, RAG answers, long-form QA), each labeled supported or unsupported by people. The official metric is balanced accuracy per dataset, averaged; 50% is chance.
First measurement pending
The benchmark script is part of the engine's source code and runs with one click once the dataset access is set up. Results are published here as measured, including the datasets where the engine does worse.
Your data is not a benchmark
Public numbers show the engine works. A quality audit shows how your own AI does on your own traffic.