REAL OUTPUT, INCLUDING THE FAILURES
What this product actually produced, without waiting for it.
Complete responses captured from real runs against the public demo fragments: every claim, every citation, which citations survived the live audit, and which thresholds passed.
Nothing is curated. No run was repeated to get a better result, and the abstentions sit beside the successes. Held-out fragments are never captured.
Choose a fragment
Loading captured dossiers…
Dossier
Pick a fragment above to read its complete evidence dossier.
How to read these
One dossier is one run, and the next run may disagree with it
Over 28 recorded runs of these five fragments, only one case gave the same verdict every time. D02 returned four different leading candidates in four runs; the dossier here is the one where it answered Bray Studios Inc., the studio rather than the film. Across all 28 runs there were 9 false-confident identifications, not the zero that a single pass recorded.
These captures predate the two fixes, deliberately: the D02 dossier shows the candidate slot holding a studio, which is exactly the defect app/works.py now rejects. Each dossier reports the gate it was actually scored under.
One thing did not move. No run, in 28, ever reached probable, because that threshold needs a human the API cannot supply. The boundary between candidates and abstain wobbles; the guarantee that the system will not assert an identity on its own does not.
Every run, case by case, with the before and after of the fixes.
An abstention is a correct answer, not a failure to produce one
On D04 the research worked: thirteen domains, the signage correctly transcribed. Then ten citations were rejected because the quoted text could not be re-found on the live page. No decisive claim survived, so the system abstained on a fragment whose answer key names a real film.
That is a real cost, kept on purpose, and it is not a guarantee: on other runs the same fragment produced Un coin de Paris (1900), which is the wrong film. The strictness reduces confident errors without eliminating them, and the study says by how much.