A benchmark score is not a deployment decision

A quality score can tell us how a model performed on a test. It cannot, by itself, tell us whether that model belongs in a sovereign, private or air-gapped production system.

That decision also depends on which artifact ran, how inputs reached it, what resources it consumed, which controls were actually enforced and whether somebody can verify the evidence later.

I built RunWitness to preserve that context around an independent benchmark—not to replace the benchmark or redefine its score.

The experiment

I ran two local Ollama document-extraction pipelines against named cases from DocuBench’s starter profile:

The tempting conclusion is that Phi 3.5 won. The evidence does not support that claim. The tracks used different documents and different modalities: images for the vision track, native extracted text for the text track. The percentages describe their own runs; they are not a controlled comparison.

That limitation is not a footnote. It is part of the result.

What the evidence adds

For each run, RunWitness preserved the benchmark’s native output and recorded:

• the exact Ollama model identity and digest;
• the selected document cases and input modality;
• per-document outcomes and aggregate benchmark result;
• wall time and scoped runtime resource observations;
• declared deployment gates and their outcomes;
• integrity hashes that allow the bundle to be checked later.

This makes a review more useful than a screenshot of a leaderboard. A reviewer can distinguish the benchmark’s finding from the deployment wrapper’s finding—and see where the evidence stops.

Do not convert declarations into claims

Both runs communicated with a local Ollama service. Therefore network isolation was not enforced. The recorded Ollama allocation is not whole-machine peak memory. The adapter process’s resident memory is not the model server’s memory. Running locally does not prove that an air gap exists.

“Air-gapped” is a property to enforce and attest, not a label to add because the endpoint is local.

A defensible evidence model separates four states:

1. Declared: what the operator says should be true.
2. Enforced: what a control actively constrained.
3. Observed: what the available instrumentation measured.
4. Unavailable: what this run could not establish.

This vocabulary prevents a common failure in enterprise AI: turning configuration intent into assurance language.

From benchmark to acceptance case

The benchmark should continue to own its data and scoring. The surrounding evidence layer should answer a different set of questions: did the intended artifact run, within the declared boundary, under the required operating constraints—and can the claim be independently checked?

That approach is benchmark-agnostic. DocuBench is the first reference integration because document extraction exposes the problem clearly, but the same pattern can wrap another evaluation tool without copying its code or taking ownership of its methodology.

The practical lesson is simple: retain the score, but make deployment approval depend on an evidence bundle. Quality, provenance, resource boundaries, isolation and integrity belong in the same review, with uncertainty made explicit.

Read the reproducible case study →

This experiment used local development hardware and a small starter profile; it is not a production capacity study or a model ranking. Personal views only. It does not describe or represent any employer, client or confidential system.