Benchmark

Measured, not claimed.

How often the engine is right, on examples labeled by people. The same scripts run on every engine change, so accuracy can only go up.

Test set ll-testset-1.0

40 hand-labeled examples

5 numeric drift, 5 fabricated entity, 5 contradiction, 8 faithful, 5 meaning error, 6 instruction breach, 3 unsupported addition, 3 clean with instructions. A hallucination means the answer contains at least one claim the source contradicts or does not support.

ConfigurationPrecisionRecallBalanced accuracyFalse alarmsFailure types foundInstruction checks
Rules only
heuristic-1.3
100%70.8%85.4%0100%100%
Rules + HHEM (production)
heuristic-1.3
100%95.8%97.9%0100%100%
Known misses, Rules only (7)
  • meaning error (meaning-1): missed hallucination
  • meaning error (meaning-2): missed hallucination
  • meaning error (meaning-3): missed hallucination
  • meaning error (meaning-4): missed hallucination
  • meaning error (meaning-5): missed hallucination
  • instruction breach (instr-4): missed hallucination
  • unsupported addition (added-2): missed hallucination
Known misses, Rules + HHEM (production) (1)
  • meaning error (meaning-5): missed hallucination

Meaning errors such as role reversals or before/after swaps share every word with the source, so word-level rules cannot see them; the HHEM model and the optional LLM judge are there for those.

Public benchmark

LLM-AggreFact

The standard public test of grounded fact-checking: real documents and claims from more than ten datasets (summaries, RAG answers, long-form QA), each labeled supported or unsupported by people. The official metric is balanced accuracy per dataset, averaged; 50% is chance.

First measurement pending

The benchmark script is part of the engine's source code and runs with one click once the dataset access is set up. Results are published here as measured, including the datasets where the engine does worse.

Your data is not a benchmark

Public numbers show the engine works. A quality audit shows how your own AI does on your own traffic.

About quality audits