Witness Graph

An agent harness where a claim without a citation is not an answer

A metric moved. The agent has to say whether the business changed or the instrument broke, name the specific fault, and cite the evidence for it. The scorer never reads the agent's summary of its own work. It reads the append-only ledger the runtime wrote while the agent ran.

What the scorer found

Every number below is the output of harness/score.py over the 10 committed run ledger(s), regenerated when this page was built. Nothing here is typed by hand.

Verdict accuracy

0.500

Did it name the right kind of failure

Root cause accuracy

0.600

Did it name the specific fault

Root cause lift

+0.500

Over the lazy agent on these same cases

Citation precision

1.000

Cited evidence that exists in the ledger

Citation recall

0.767

Of the evidence the answer key requires

Lucky guess rate

0.400

Right answer, evidence never looked at

Unapproved effects

Writes that skipped the approval gate

Out-of-scope calls refused

Agent reached outside its allowlist and the runtime blocked it

Out-of-scope calls executed

Refusals that did not hold. Must be zero

10 of 11 cases have run, so read the lift with that in mind. The lift card above compares against the lazy agent on these same cases, rather than against a baseline drawn from the whole corpus, which would be a comparison between two different samples. Against the full corpus the baseline is 0.545 on the verdict and 0.091 on the root cause.

The runs themselves

Each row opens the full run page: which of the eleven nodes fired, every tool call, every witness the agent gathered, and the verdict set against the answer key. These are projections of the ledger, so a page cannot flatter a run that went badly.

Case	Difficulty	Verdict	Root cause	Citation recall	Lucky guess	Tool calls
MTI-001	easy	correct	correct	0.50	yes	35
MTI-002	medium	correct	correct	1.00	no	41
MTI-003	medium	correct	correct	1.00	no	33
MTI-004	hard	correct	correct	0.50	yes	41
MTI-005	medium	wrong	wrong	1.00	no	33
MTI-006	hard	correct	correct	0.67	yes	38
MTI-007	hard	wrong	correct	0.67	yes	39
MTI-008	medium	wrong	wrong	0.67	no	41
MTI-009	medium	wrong	wrong	1.00	no	35
MTI-011	hard	wrong	wrong	0.67	no	41

Reproduce it

The faults are seeded, so the answer key is decided by construction rather than by our opinion. Scoring runs against the committed ledgers with no API key and no DataHub instance.

git clone https://github.com/HDPark95/witness-graph
cd witness-graph
python3 harness/warehouse.py --all --out warehouse/
python3 harness/score.py --cases cases/ --runs runs/
python3 harness/check_submission.py --warehouses warehouse/

The last command is the one worth running. It fails the build when the repository is not in a state a judge could evaluate: a node declaring a tool that is not registered, a committed ledger that answers its own case wrongly, a case corpus that no longer regenerates identically, an absolute path leaking into a ledger.

What is deliberately visible

Lucky guesses are reported, not hidden: a run that names the right fault without ever citing the evidence for it is marked, because the difference between knowing and guessing is the entire point. Refused tool calls are reported the same way. A run that reaches for a tool its node never declared gets stopped by the runtime and logged, and that log is what lets the scorer claim zero unapproved effects without taking the model's word for it.

Source, benchmark and scorer: https://github.com/HDPark95/witness-graph · Apache-2.0 · Full scorer output: summary.json