The knowledge argument: where logic lives

When an answer depends on a business definition, the definition has to live somewhere a machine can apply it.

The bench asks one question three ways: list every high-risk vendor. A vendor is high-risk when it operates in a flagged region, processes sensitive data and holds a contract worth more than one million euro. The three ways share one harness. Only the step that writes the query differs.

On scenario A the pure LLM returns 5, the property graph returns 3 and the semantic graph returns 4, which is the correct answer.

Why the three answers differ

Pure LLM: 5, too many

It reads the documents and is misled by an amendment that was cancelled but never removed. It over-flags.

Property graph: 3, too few

Its high-risk label was worked out in advance, before a later amendment lifted a contract over the line. The label went stale, so it misses a vendor.

Semantic graph: 4, correct

The definition is written once as a rule, and a reasoner applies it to the current facts every time it is asked. Nothing is stale and nothing is guessed.

The two naive methods are wrong in opposite directions. Only the reasoner reflects the reconciled current truth.

Asserted against inferred

The bench can show the proof. In the data as entered, no vendor is recorded as high-risk: 0 asserted. With the rule applied, 4 are inferred. The second scenario, a legal change and the customers it affects, shows the same shape: 0 asserted and 2 inferred.

The ladder, and a caution

This does not mean every capability needs a semantic graph. The architecture offers five rungs, from a versioned prompt up to a formal semantic model, and says to start on the lowest rung that works. The bench climbs to the top because this question demands it, and it says so openly.

These figures are stated here as a narrative. The site does not run the comparison. The bench does, in a demonstration: see Demo and contact.