Every automated system makes mistakes. The question is whether its users can see them. A tool that reports only its successes is asking for trust it has not demonstrated. A tool that shows its failures is giving users what they need to decide how far to rely on it.
Accuracy claims need evidence too
"99% accurate" is a claim like any other. Accurate on what? Measured by whom? Against which reference? An accuracy figure without its test cases is exactly the kind of unsupported statement a fact-checker should reject. So Honest Lens applies its own standard to itself: the Self-Audit page lists individual cases, the reference verdict, the engine's verdict, and a note on each, including the disagreements.
Failure modes, not just failure rates
A single error rate says little about when to be careful. Patterns of failure are more useful. The failure modes an AI fact-checker should expect, and should tell its users about, include:
- Irony and local idiom. Sarcasm and regional expressions can be read as literal claims, especially across languages.
- Nested statistics. When a source separates a sub-category from a total, a model can confuse one for the other.
- Missing records. When official documents are not digitised or public, the only honest answer is Insufficient Evidence.
- Recent events. Anything after a model's training data is unknown to it unless supplied.
Why "I don't know" counts as success
An engine that declines to judge when the evidence is thin is not failing. It is working as designed. That is why the self-audit tracks honest refusals as their own category rather than counting them as errors or hiding them.
What users can do
- Read the explanation, not only the verdict.
- Check the sources, especially for high-stakes claims.
- Use the peer-review buttons on each claim to flag evaluations that seem too strict or incomplete.
The principle
Transparency is not a feature to add once a system works well. It is how anyone, including the people who build it, can find out whether it works at all.